2021-05-25T01:05:30.336 INFO:root:teuthology version: 1.1.0-d92ec140
2021-05-25T01:05:30.441 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging
2021-05-25T01:05:30.442 INFO:teuthology.lock.ops:Updating [ubuntu@smithi074.front.sepia.ceph.com]: reset os type and version on server
2021-05-25T01:05:30.442 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server
2021-05-25T01:05:30.459 INFO:teuthology.lock.ops:Node 'ubuntu@smithi074.front.sepia.ceph.com' reimaging is complete
2021-05-25T01:05:30.461 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging
2021-05-25T01:05:30.461 INFO:teuthology.lock.ops:Updating [ubuntu@smithi183.front.sepia.ceph.com]: reset os type and version on server
2021-05-25T01:05:30.461 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server
2021-05-25T01:05:30.480 INFO:teuthology.lock.ops:Node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging is complete
2021-05-25T01:05:30.677 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of ubuntu 20.04
2021-05-25T01:05:30.691 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of ubuntu 20.04
2021-05-25T01:05:31.054 INFO:teuthology.orchestra.console:Power off smithi074
2021-05-25T01:05:31.107 INFO:teuthology.orchestra.console:Power off smithi183
2021-05-25T01:05:39.445 INFO:teuthology.orchestra.console:Power off for smithi074 completed
2021-05-25T01:05:39.473 INFO:teuthology.orchestra.console:Power off for smithi183 completed
2021-05-25T01:05:39.548 INFO:teuthology.orchestra.console:Power on smithi074
2021-05-25T01:05:39.576 INFO:teuthology.orchestra.console:Power on smithi183
2021-05-25T01:05:47.966 INFO:teuthology.orchestra.console:Power on for smithi183 completed
2021-05-25T01:05:47.976 INFO:teuthology.orchestra.console:Power on for smithi074 completed
2021-05-25T01:05:48.067 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish
2021-05-25T01:05:48.078 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish
2021-05-25T01:10:45.718 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-25T01:10:45.745 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-05-25T01:10:46.188 INFO:teuthology.provision.fog.smithi183:Node is ready
2021-05-25T01:10:46.200 INFO:teuthology.provision.fog.smithi074:Node is ready
2021-05-25T01:10:46.207 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com
2021-05-25T01:10:46.235 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183.front.sepia.ceph.com
2021-05-25T01:10:46.243 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.183 smithi183.front.sepia.ceph.com smithi183
2021-05-25T01:10:46.257 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074
2021-05-25T01:10:46.417 INFO:teuthology.provision.fog.smithi183:Deploy complete!
2021-05-25T01:10:46.450 INFO:teuthology.provision.fog.smithi074:Deploy complete!
2021-05-25T01:10:46.605 INFO:teuthology.lock.ops:Checking smithi074.front.sepia.ceph.com
2021-05-25T01:10:46.606 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-25T01:10:46.624 INFO:teuthology.lock.ops:Checking smithi183.front.sepia.ceph.com
2021-05-25T01:10:46.624 INFO:teuthology.lock.ops:New key found. Updating...
2021-05-25T01:10:46.648 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: set os type and version on server
2021-05-25T01:10:46.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-25T01:10:46.822 INFO:teuthology.orchestra.run.smithi074.stdout:x86_64
2021-05-25T01:10:46.872 INFO:teuthology.orchestra.run.smithi074.stdout:NAME="Ubuntu"
2021-05-25T01:10:46.872 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-25T01:10:46.873 INFO:teuthology.orchestra.run.smithi074.stdout:ID=ubuntu
2021-05-25T01:10:46.873 INFO:teuthology.orchestra.run.smithi074.stdout:ID_LIKE=debian
2021-05-25T01:10:46.873 INFO:teuthology.orchestra.run.smithi074.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-25T01:10:46.873 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_ID="20.04"
2021-05-25T01:10:46.873 INFO:teuthology.orchestra.run.smithi074.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-25T01:10:46.874 INFO:teuthology.orchestra.run.smithi074.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-25T01:10:46.874 INFO:teuthology.orchestra.run.smithi074.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-25T01:10:46.874 INFO:teuthology.orchestra.run.smithi074.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-25T01:10:46.874 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_CODENAME=focal
2021-05-25T01:10:46.875 INFO:teuthology.orchestra.run.smithi074.stdout:UBUNTU_CODENAME=focal
2021-05-25T01:10:46.876 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server
2021-05-25T01:10:46.893 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: set os type and version on server
2021-05-25T01:10:46.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-25T01:10:47.060 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64
2021-05-25T01:10:47.107 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Ubuntu"
2021-05-25T01:10:47.107 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="20.04.2 LTS (Focal Fossa)"
2021-05-25T01:10:47.108 INFO:teuthology.orchestra.run.smithi183.stdout:ID=ubuntu
2021-05-25T01:10:47.108 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE=debian
2021-05-25T01:10:47.108 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Ubuntu 20.04.2 LTS"
2021-05-25T01:10:47.108 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="20.04"
2021-05-25T01:10:47.108 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.ubuntu.com/"
2021-05-25T01:10:47.109 INFO:teuthology.orchestra.run.smithi183.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2021-05-25T01:10:47.109 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2021-05-25T01:10:47.109 INFO:teuthology.orchestra.run.smithi183.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2021-05-25T01:10:47.109 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_CODENAME=focal
2021-05-25T01:10:47.110 INFO:teuthology.orchestra.run.smithi183.stdout:UBUNTU_CODENAME=focal
2021-05-25T01:10:47.110 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server
2021-05-25T01:10:47.200 INFO:teuthology.dispatcher.supervisor:Running job 6132603
2021-05-25T01:10:47.200 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-05-25_01:02:17-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6132603 --name xiubli-2021-05-25_01:02:17-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi --description fs:functional:/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{centos_8.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/ubuntu_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore/bluestore-bitmap.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/mds-full.yaml} -- /home/teuthworker/archive/xiubli-2021-05-25_01:02:17-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6132603/orig.config.yaml
2021-05-25T01:10:47.204 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-05-25_01:02:17-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6132603
2021-05-25T01:10:47.205 INFO:teuthology.dispatcher.supervisor:Job PID: 14260
2021-05-25T01:10:47.205 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-05-25T03:40:56.234 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/xiubli-2021-05-25_01:02:17-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/jobs/ failed with status 500:
Internal Server Error
Internal Server Error
2021-05-25T13:11:34.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-05-25T13:11:34.738 INFO:teuthology.kill:Killing Pids: {14260}
2021-05-25T13:11:35.767 INFO:teuthology.kill:Nuking machines: ['smithi074', 'smithi183']
2021-05-25T13:11:36.651 INFO:teuthology.kill:2021-05-25 13:11:36,650.650 INFO:teuthology.nuke:targets:
2021-05-25T13:11:36.651 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXzuJeww6nvL9QD+C2g0U0DaA95aiU54AVctMFqpkVa5eWL1ydSE8+HzISYW2pDkoN3Gt8jZy8L0QBPtCgJ0/hbrOohTyD6GTnDMWyeC/MaXfNRQvxIJWOXo5n4K5dnr4tcGrD2wMTqz7RkmEtue0hSXUHlqyHlrIplqJFdmsN4R7R/4pcmt7sy8YKUQUZlK/MS4jUXQjgoEpWl9QaoZkfRkUsygzQ3HD6ZuWYKp7dagEsAzqlswHoY1QNMdXWmcIOLzX/q6DlUOOidrGbs1W/rs+iyMqf1PoKLOkCinoePxvQLs4ErJKXzcs1ChHLdUmWNzX75moPtlVsXFIbbnOhDgQkZvR3Jcc2Cq8TdqlgC4hJG+GQxRXS05WnWymsDuLgvRgrmMhY5oa7YzJVQr4CUNaPL/vGgayJgxg8s2FOevlYy/Je1Q97Ph3gldQUkdyF5Bz2VwWFY4f+Vyw5Byo7GOLUqrbn7Z+BO5SekbNK9l5UYOg1AGhhMN3+i6uwT3c=
2021-05-25T13:11:36.652 INFO:teuthology.kill: smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqxmAk/+EJARaG1JOK+AUNO6+NutdYhfQsOOvOipUfDaM4WsUsaQ2A5ZzN3NM2coUNRKuesOMUfxspuL3hZnj4EQjWdAE3Fp0V25Xi11HhFIXqVLV9Nzh6bjHd5RDO64Fts2ejijWS8Uk2d9Nqy6ymTovpbEGYLawmSfxybJpcjHtOm26EEA+SJ1YcEVAM4T5QthPBwvt9ukpsy0nGrZbJaDzipa8V94aS1uzV9qgynJF2AAYtXFnrN82NXuG0OGD2HInewrrJ2T7cxB501IPS+RQne3bGThe6w8cSxCmLq993AIk+YgP1GCoX5r+rLiWtSLkSOWxoOr0MBmYdmNPhoGzzhBflSKLoI+6Q/LdhF6gZVvHPQb0rfHMRKpss9X/hUOthWOkqZd4kWcAXp9QPIPwx5N7tXfVz9oprYomaVt1JgyNJwqyf33VmPkWlYMkxPnvxZ3vLb6c8OqNp2VpamJNQwX7u4B34JnFWUyupsmsBd7I6zpjxnpFo+aekQRk=
2021-05-25T13:11:36.652 INFO:teuthology.kill:2021-05-25 13:11:36,651.651 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-25T13:11:36.653 INFO:teuthology.kill:2021-05-25 13:11:36,653.653 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-25T13:11:36.714 INFO:teuthology.kill:2021-05-25 13:11:36,714.714 INFO:teuthology.orchestra.console:Power off smithi183
2021-05-25T13:11:36.726 INFO:teuthology.kill:2021-05-25 13:11:36,726.726 INFO:teuthology.orchestra.console:Power off smithi074
2021-05-25T13:11:49.194 INFO:teuthology.kill:2021-05-25 13:11:49,193.193 INFO:teuthology.orchestra.console:Power off for smithi183 completed
2021-05-25T13:11:49.199 INFO:teuthology.kill:2021-05-25 13:11:49,199.199 INFO:teuthology.orchestra.console:Power off for smithi074 completed
2021-05-25T13:11:49.415 INFO:teuthology.task.internal:roles: ubuntu@smithi074.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2']
2021-05-25T13:11:49.416 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']
2021-05-25T13:11:49.416 INFO:teuthology.misc:Compressing logs...
2021-05-25T13:11:49.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-05-25T13:12:07.883 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
teuth_config.archive_base, job_config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
compress_logs(ctx, log_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/misc.py", line 1378, in compress_logs
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/remote.py", line 509, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/run.py", line 434, in run
transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-05-25T13:12:07.949 INFO:teuthology.kill:No teuthology processes running
2021-05-25T13:12:08.954 INFO:teuthology.kill:Nuking machines: ['smithi074', 'smithi183']
2021-05-25T13:12:09.813 INFO:teuthology.kill:2021-05-25 13:12:09,812.812 INFO:teuthology.nuke:targets:
2021-05-25T13:12:09.814 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXzuJeww6nvL9QD+C2g0U0DaA95aiU54AVctMFqpkVa5eWL1ydSE8+HzISYW2pDkoN3Gt8jZy8L0QBPtCgJ0/hbrOohTyD6GTnDMWyeC/MaXfNRQvxIJWOXo5n4K5dnr4tcGrD2wMTqz7RkmEtue0hSXUHlqyHlrIplqJFdmsN4R7R/4pcmt7sy8YKUQUZlK/MS4jUXQjgoEpWl9QaoZkfRkUsygzQ3HD6ZuWYKp7dagEsAzqlswHoY1QNMdXWmcIOLzX/q6DlUOOidrGbs1W/rs+iyMqf1PoKLOkCinoePxvQLs4ErJKXzcs1ChHLdUmWNzX75moPtlVsXFIbbnOhDgQkZvR3Jcc2Cq8TdqlgC4hJG+GQxRXS05WnWymsDuLgvRgrmMhY5oa7YzJVQr4CUNaPL/vGgayJgxg8s2FOevlYy/Je1Q97Ph3gldQUkdyF5Bz2VwWFY4f+Vyw5Byo7GOLUqrbn7Z+BO5SekbNK9l5UYOg1AGhhMN3+i6uwT3c=
2021-05-25T13:12:09.814 INFO:teuthology.kill: smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqxmAk/+EJARaG1JOK+AUNO6+NutdYhfQsOOvOipUfDaM4WsUsaQ2A5ZzN3NM2coUNRKuesOMUfxspuL3hZnj4EQjWdAE3Fp0V25Xi11HhFIXqVLV9Nzh6bjHd5RDO64Fts2ejijWS8Uk2d9Nqy6ymTovpbEGYLawmSfxybJpcjHtOm26EEA+SJ1YcEVAM4T5QthPBwvt9ukpsy0nGrZbJaDzipa8V94aS1uzV9qgynJF2AAYtXFnrN82NXuG0OGD2HInewrrJ2T7cxB501IPS+RQne3bGThe6w8cSxCmLq993AIk+YgP1GCoX5r+rLiWtSLkSOWxoOr0MBmYdmNPhoGzzhBflSKLoI+6Q/LdhF6gZVvHPQb0rfHMRKpss9X/hUOthWOkqZd4kWcAXp9QPIPwx5N7tXfVz9oprYomaVt1JgyNJwqyf33VmPkWlYMkxPnvxZ3vLb6c8OqNp2VpamJNQwX7u4B34JnFWUyupsmsBd7I6zpjxnpFo+aekQRk=
2021-05-25T13:12:09.833 INFO:teuthology.kill:2021-05-25 13:12:09,833.833 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-25T13:12:09.836 INFO:teuthology.kill:2021-05-25 13:12:09,835.835 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-05-25T13:12:09.889 INFO:teuthology.kill:2021-05-25 13:12:09,889.889 INFO:teuthology.orchestra.console:Power off smithi074
2021-05-25T13:12:09.900 INFO:teuthology.kill:2021-05-25 13:12:09,900.900 INFO:teuthology.orchestra.console:Power off smithi183
2021-05-25T13:12:10.028 INFO:teuthology.kill:2021-05-25 13:12:10,028.028 INFO:teuthology.orchestra.console:Power off for smithi074 completed
2021-05-25T13:12:10.037 INFO:teuthology.kill:2021-05-25 13:12:10,037.037 INFO:teuthology.orchestra.console:Power off for smithi183 completed
2021-05-25T13:12:10.166 INFO:teuthology.kill:2021-05-25 13:12:10,165.165 INFO:teuthology.lock.ops:unlocked smithi074.front.sepia.ceph.com
2021-05-25T13:12:10.175 INFO:teuthology.kill:2021-05-25 13:12:10,175.175 INFO:teuthology.lock.ops:unlocked smithi183.front.sepia.ceph.com
2021-05-25T13:14:10.443 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-05-25T13:14:10.550 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-05-25T13:14:10.550 INFO:teuthology.nuke:Checking targets against current locks
2021-05-25T13:14:10.618 INFO:teuthology.nuke:Not nuking smithi074.front.sepia.ceph.com because description doesn't match
2021-05-25T13:14:10.618 INFO:teuthology.nuke:Not nuking smithi183.front.sepia.ceph.com because description doesn't match