2021-05-27T08:08:29.332 INFO:root:teuthology version: 1.1.0-0c70841a 2021-05-27T08:08:29.425 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging 2021-05-27T08:08:29.426 INFO:teuthology.lock.ops:Updating [ubuntu@smithi062.front.sepia.ceph.com]: reset os type and version on server 2021-05-27T08:08:29.427 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-05-27T08:08:29.440 INFO:teuthology.lock.ops:Node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging is complete 2021-05-27T08:08:29.441 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging 2021-05-27T08:08:29.441 INFO:teuthology.lock.ops:Updating [ubuntu@smithi170.front.sepia.ceph.com]: reset os type and version on server 2021-05-27T08:08:29.441 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-05-27T08:08:29.456 INFO:teuthology.lock.ops:Node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging is complete 2021-05-27T08:08:29.644 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of centos 8.3 2021-05-27T08:08:29.652 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.3 2021-05-27T08:08:29.962 INFO:teuthology.orchestra.console:Power off smithi062 2021-05-27T08:08:29.976 INFO:teuthology.orchestra.console:Power off smithi170 2021-05-27T08:08:38.334 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-05-27T08:08:38.339 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-05-27T08:08:38.435 INFO:teuthology.orchestra.console:Power on smithi062 2021-05-27T08:08:38.442 INFO:teuthology.orchestra.console:Power on smithi170 2021-05-27T08:08:46.796 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2021-05-27T08:08:46.801 INFO:teuthology.orchestra.console:Power on for smithi170 completed 2021-05-27T08:08:46.897 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2021-05-27T08:08:46.902 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish 2021-05-27T08:11:18.470 ERROR:teuthology.orchestra.connection:Error authenticating with smithi170.front.sepia.ceph.com: Authentication failed. 2021-05-27T08:12:59.143 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-27T08:13:14.485 INFO:teuthology.provision.fog.smithi170:Node is ready 2021-05-27T08:13:14.544 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com 2021-05-27T08:13:14.601 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170 2021-05-27T08:13:14.962 INFO:teuthology.provision.fog.smithi170:Deploy complete! 2021-05-27T08:13:25.286 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-27T08:13:25.603 INFO:teuthology.provision.fog.smithi062:Node is ready 2021-05-27T08:13:25.617 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2021-05-27T08:13:25.673 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2021-05-27T08:13:25.995 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2021-05-27T08:13:26.109 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2021-05-27T08:13:26.110 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-27T08:13:26.130 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com 2021-05-27T08:13:26.131 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-27T08:13:26.144 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2021-05-27T08:13:26.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T08:13:26.426 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2021-05-27T08:13:26.481 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Linux" 2021-05-27T08:13:26.481 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8" 2021-05-27T08:13:26.481 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2021-05-27T08:13:26.481 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2021-05-27T08:13:26.482 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8" 2021-05-27T08:13:26.482 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2021-05-27T08:13:26.482 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-27T08:13:26.482 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2021-05-27T08:13:26.482 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-27T08:13:26.483 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2021-05-27T08:13:26.483 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-27T08:13:26.483 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-27T08:13:26.483 INFO:teuthology.orchestra.run.smithi062.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-27T08:13:26.484 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-05-27T08:13:26.499 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server 2021-05-27T08:13:26.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T08:13:26.783 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2021-05-27T08:13:26.837 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Linux" 2021-05-27T08:13:26.837 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8" 2021-05-27T08:13:26.837 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2021-05-27T08:13:26.838 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2021-05-27T08:13:26.838 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8" 2021-05-27T08:13:26.838 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2021-05-27T08:13:26.838 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Linux 8" 2021-05-27T08:13:26.839 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2021-05-27T08:13:26.839 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-27T08:13:26.839 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2021-05-27T08:13:26.839 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-27T08:13:26.839 INFO:teuthology.orchestra.run.smithi170.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-27T08:13:26.840 INFO:teuthology.orchestra.run.smithi170.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-27T08:13:26.840 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-05-27T08:13:26.915 INFO:teuthology.dispatcher.supervisor:Running job 6137787 2021-05-27T08:13:26.915 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2021-05-27_08:08:15-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6137787 --name xiubli-2021-05-27_08:08:15-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi --description fs:functional:/{begin.yaml clusters/1a3s-mds-4c-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} distro/{rhel_8.yaml} mount/kclient/{mount.yaml overrides/{distro/testing/{flavor/centos_latest.yaml k-testing.yaml} ms-die-on-skipped.yaml}} objectstore/bluestore-ec-root.yaml overrides/{frag_enable.yaml no_client_pidfile.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/mds-full.yaml} -- /home/teuthworker/archive/xiubli-2021-05-27_08:08:15-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6137787/orig.config.yaml 2021-05-27T08:13:26.918 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2021-05-27_08:08:15-fs:functional:-wip-lxb-20210524-45434-pg-num-distro-basic-smithi/6137787 2021-05-27T08:13:26.919 INFO:teuthology.dispatcher.supervisor:Job PID: 3149 2021-05-27T08:13:26.919 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-27T20:13:41.323 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-27T20:13:41.536 INFO:teuthology.kill:Killing Pids: {3149} 2021-05-27T20:13:43.477 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi170'] 2021-05-27T20:13:44.446 INFO:teuthology.kill:2021-05-27 20:13:44,446.446 INFO:teuthology.nuke:targets: 2021-05-27T20:13:44.447 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA9IJvw6eoZkKiK9mupc/1nVdjZv5fzXP26/8996S9a2inETeRJ06WMpm8/Y8XzWXI3VTTe9Lcp+0Kyfcm8afG9LInhExMG9Ccui6+sQomPDhhCclmPnfzoeCRszI8JiOcS/KgwKvrqp57dvqZMkNpMFCrGIfUjWOo5nFA62qLApwjsLsGGfkcx9TKExkTsmKq5Yghf9mjkoshtQvhf1ZIecCQQViBlA1wCGr9SwTNSycc4HLGd9dBLbSYjArSTVBjVtXYYBALHvJLK1UZnE/HDRQ11/5dQEqwc+1BYKMge8ueiugceRRz4ghJKnekhJdEfYgUp0xVXBlf1SWeqrpPLJ8ZJck5t3wcTtspBIO+vaZLHiPeIYAWD7YKwDV4a2DACXwmxMmhcrthrqP3+43nWaqj0ecxdPp05wvHQFF+gyTcemUds3ZAsTd/Lo2MpEoa+sdeeqESlow0kucVbmCaYrkbPMEfZZo/T3aauIwk5nv1fkGrRg9skGaOWwTRfKk= 2021-05-27T20:13:44.448 INFO:teuthology.kill: smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNZ3bzzecBcKGLt+JHSeaaK20Z4iDqIdxD7pCqf3DjOBcnFU/Qx7OBq1/uiwV+gAirGLPEmU59u3Dl/whkjsKRjdZFpFOn2hs5vy7ir8vR+72KzyxfgHuri2UfhsNKwK5qlToLuNtOdbSdscm+phv0P1INMxdYUfIbRs9u6ExbbIaWm6rorVvu1Ltu+CrV9HpGcWSyypkHxaR7kaC4yzYzuzo9q5muXO/TCmV6QZ/TsOocayANmblkNl5aQnw9l72hK0meGzITvGp6nbDqu9c3iCdfzRo8TbBi/zD+ueXelm83t+Gvfl8Q8CAg2xj2q7W+4Zvzl5R1wn4iL+9bWMz8KSSWiuUe481Aahd0Y0fb9s48bC9FVHvnJGFiZEkL0V5ChQh3UVaFW7WzF99nBvMVAelBv9DE4DKI2uv5UGDOvekCkr/X4ytAzwX+6ZJJH8btP6LnGvKxTVbOXrornnyqFu0Aoz9X1G51uCRQ1EpgxtvbAvnAiyIKstbf5LclU0= 2021-05-27T20:13:44.449 INFO:teuthology.kill:2021-05-27 20:13:44,446.446 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T20:13:44.449 INFO:teuthology.kill:2021-05-27 20:13:44,449.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T20:13:44.507 INFO:teuthology.kill:2021-05-27 20:13:44,507.507 INFO:teuthology.orchestra.console:Power off smithi170 2021-05-27T20:13:44.522 INFO:teuthology.kill:2021-05-27 20:13:44,521.521 INFO:teuthology.orchestra.console:Power off smithi062 2021-05-27T20:13:52.878 INFO:teuthology.kill:2021-05-27 20:13:52,878.878 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-05-27T20:13:56.997 INFO:teuthology.kill:2021-05-27 20:13:56,996.996 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-05-27T20:13:57.268 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2021-05-27T20:13:57.268 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2021-05-27T20:13:57.269 INFO:teuthology.misc:Compressing logs... 2021-05-27T20:13:57.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T20:14:57.273 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0c70841acf4fb31ca05cdbebbd295620aac53ce4/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-27T20:14:57.342 INFO:teuthology.kill:No teuthology processes running 2021-05-27T20:14:58.403 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi170'] 2021-05-27T20:14:59.327 INFO:teuthology.kill:2021-05-27 20:14:59,327.327 INFO:teuthology.nuke:targets: 2021-05-27T20:14:59.328 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDA9IJvw6eoZkKiK9mupc/1nVdjZv5fzXP26/8996S9a2inETeRJ06WMpm8/Y8XzWXI3VTTe9Lcp+0Kyfcm8afG9LInhExMG9Ccui6+sQomPDhhCclmPnfzoeCRszI8JiOcS/KgwKvrqp57dvqZMkNpMFCrGIfUjWOo5nFA62qLApwjsLsGGfkcx9TKExkTsmKq5Yghf9mjkoshtQvhf1ZIecCQQViBlA1wCGr9SwTNSycc4HLGd9dBLbSYjArSTVBjVtXYYBALHvJLK1UZnE/HDRQ11/5dQEqwc+1BYKMge8ueiugceRRz4ghJKnekhJdEfYgUp0xVXBlf1SWeqrpPLJ8ZJck5t3wcTtspBIO+vaZLHiPeIYAWD7YKwDV4a2DACXwmxMmhcrthrqP3+43nWaqj0ecxdPp05wvHQFF+gyTcemUds3ZAsTd/Lo2MpEoa+sdeeqESlow0kucVbmCaYrkbPMEfZZo/T3aauIwk5nv1fkGrRg9skGaOWwTRfKk= 2021-05-27T20:14:59.328 INFO:teuthology.kill: smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHNZ3bzzecBcKGLt+JHSeaaK20Z4iDqIdxD7pCqf3DjOBcnFU/Qx7OBq1/uiwV+gAirGLPEmU59u3Dl/whkjsKRjdZFpFOn2hs5vy7ir8vR+72KzyxfgHuri2UfhsNKwK5qlToLuNtOdbSdscm+phv0P1INMxdYUfIbRs9u6ExbbIaWm6rorVvu1Ltu+CrV9HpGcWSyypkHxaR7kaC4yzYzuzo9q5muXO/TCmV6QZ/TsOocayANmblkNl5aQnw9l72hK0meGzITvGp6nbDqu9c3iCdfzRo8TbBi/zD+ueXelm83t+Gvfl8Q8CAg2xj2q7W+4Zvzl5R1wn4iL+9bWMz8KSSWiuUe481Aahd0Y0fb9s48bC9FVHvnJGFiZEkL0V5ChQh3UVaFW7WzF99nBvMVAelBv9DE4DKI2uv5UGDOvekCkr/X4ytAzwX+6ZJJH8btP6LnGvKxTVbOXrornnyqFu0Aoz9X1G51uCRQ1EpgxtvbAvnAiyIKstbf5LclU0= 2021-05-27T20:14:59.348 INFO:teuthology.kill:2021-05-27 20:14:59,348.348 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T20:14:59.349 INFO:teuthology.kill:2021-05-27 20:14:59,349.349 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T20:14:59.406 INFO:teuthology.kill:2021-05-27 20:14:59,405.405 INFO:teuthology.orchestra.console:Power off smithi170 2021-05-27T20:14:59.419 INFO:teuthology.kill:2021-05-27 20:14:59,418.418 INFO:teuthology.orchestra.console:Power off smithi062 2021-05-27T20:14:59.549 INFO:teuthology.kill:2021-05-27 20:14:59,548.548 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-05-27T20:14:59.560 INFO:teuthology.kill:2021-05-27 20:14:59,560.560 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-05-27T20:14:59.686 INFO:teuthology.kill:2021-05-27 20:14:59,685.685 INFO:teuthology.lock.ops:unlocked smithi170.front.sepia.ceph.com 2021-05-27T20:14:59.696 INFO:teuthology.kill:2021-05-27 20:14:59,696.696 INFO:teuthology.lock.ops:unlocked smithi062.front.sepia.ceph.com 2021-05-27T20:16:59.872 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-05-27T20:16:59.987 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-05-27T20:16:59.988 INFO:teuthology.nuke:Checking targets against current locks 2021-05-27T20:17:00.056 INFO:teuthology.nuke:Not nuking smithi170.front.sepia.ceph.com because description doesn't match 2021-05-27T20:17:00.056 INFO:teuthology.nuke:Not nuking smithi062.front.sepia.ceph.com because description doesn't match