2021-04-04T06:59:22.899 INFO:root:teuthology version: 1.1.0-85d61eae 2021-04-04T06:59:23.160 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi096.front.sepia.ceph.com' reimaging 2021-04-04T06:59:23.161 INFO:teuthology.lock.ops:Updating [ubuntu@smithi096.front.sepia.ceph.com]: reset os type and version on server 2021-04-04T06:59:23.161 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2021-04-04T06:59:23.177 INFO:teuthology.lock.ops:Node 'ubuntu@smithi096.front.sepia.ceph.com' reimaging is complete 2021-04-04T06:59:23.177 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging 2021-04-04T06:59:23.178 INFO:teuthology.lock.ops:Updating [ubuntu@smithi097.front.sepia.ceph.com]: reset os type and version on server 2021-04-04T06:59:23.178 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2021-04-04T06:59:23.217 INFO:teuthology.lock.ops:Node 'ubuntu@smithi097.front.sepia.ceph.com' reimaging is complete 2021-04-04T06:59:23.411 INFO:teuthology.provision.fog.smithi096:Scheduling deploy of centos 8.2 2021-04-04T06:59:23.435 INFO:teuthology.provision.fog.smithi097:Scheduling deploy of centos 8.2 2021-04-04T06:59:23.790 INFO:teuthology.orchestra.console:Power off smithi096 2021-04-04T06:59:23.831 INFO:teuthology.orchestra.console:Power off smithi097 2021-04-04T06:59:32.247 INFO:teuthology.orchestra.console:Power off for smithi096 completed 2021-04-04T06:59:32.262 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-04-04T06:59:32.349 INFO:teuthology.orchestra.console:Power on smithi096 2021-04-04T06:59:32.369 INFO:teuthology.orchestra.console:Power on smithi097 2021-04-04T06:59:40.790 INFO:teuthology.orchestra.console:Power on for smithi096 completed 2021-04-04T06:59:40.891 INFO:teuthology.provision.fog.smithi096:Waiting for deploy to finish 2021-04-04T06:59:40.895 INFO:teuthology.orchestra.console:Power on for smithi097 completed 2021-04-04T06:59:40.996 INFO:teuthology.provision.fog.smithi097:Waiting for deploy to finish 2021-04-04T07:04:13.008 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-04T07:04:13.034 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-04-04T07:04:13.323 INFO:teuthology.provision.fog.smithi096:Node is ready 2021-04-04T07:04:13.339 INFO:teuthology.provision.fog.smithi097:Node is ready 2021-04-04T07:04:13.343 INFO:teuthology.orchestra.run.smithi096.stdout:smithi096.front.sepia.ceph.com 2021-04-04T07:04:13.355 INFO:teuthology.orchestra.run.smithi097.stdout:smithi097.front.sepia.ceph.com 2021-04-04T07:04:13.398 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.15.96 smithi096.front.sepia.ceph.com smithi096 2021-04-04T07:04:13.412 INFO:teuthology.orchestra.run.smithi097.stdout:172.21.15.97 smithi097.front.sepia.ceph.com smithi097 2021-04-04T07:04:13.737 INFO:teuthology.provision.fog.smithi096:Deploy complete! 2021-04-04T07:04:13.760 INFO:teuthology.provision.fog.smithi097:Deploy complete! 2021-04-04T07:04:13.920 INFO:teuthology.lock.ops:Checking smithi096.front.sepia.ceph.com 2021-04-04T07:04:13.922 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-04T07:04:13.953 INFO:teuthology.lock.ops:Checking smithi097.front.sepia.ceph.com 2021-04-04T07:04:13.954 INFO:teuthology.lock.ops:New key found. Updating... 2021-04-04T07:04:13.973 INFO:teuthology.lock.ops:Updating [smithi096.front.sepia.ceph.com]: set os type and version on server 2021-04-04T07:04:13.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T07:04:14.279 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2021-04-04T07:04:14.335 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Linux" 2021-04-04T07:04:14.336 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="8 (Core)" 2021-04-04T07:04:14.336 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2021-04-04T07:04:14.337 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2021-04-04T07:04:14.337 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="8" 2021-04-04T07:04:14.337 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el8" 2021-04-04T07:04:14.338 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-04-04T07:04:14.338 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2021-04-04T07:04:14.339 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-04T07:04:14.339 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://www.centos.org/" 2021-04-04T07:04:14.340 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-04T07:04:14.340 INFO:teuthology.orchestra.run.smithi096.stdout: 2021-04-04T07:04:14.340 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-04T07:04:14.341 INFO:teuthology.orchestra.run.smithi096.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-04T07:04:14.341 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-04-04T07:04:14.342 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-04-04T07:04:14.342 INFO:teuthology.orchestra.run.smithi096.stdout: 2021-04-04T07:04:14.344 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2021-04-04T07:04:14.366 INFO:teuthology.lock.ops:Updating [smithi097.front.sepia.ceph.com]: set os type and version on server 2021-04-04T07:04:14.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T07:04:14.686 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2021-04-04T07:04:14.741 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Linux" 2021-04-04T07:04:14.742 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="8 (Core)" 2021-04-04T07:04:14.742 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2021-04-04T07:04:14.743 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2021-04-04T07:04:14.743 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="8" 2021-04-04T07:04:14.744 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el8" 2021-04-04T07:04:14.744 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-04-04T07:04:14.744 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2021-04-04T07:04:14.745 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-04-04T07:04:14.745 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://www.centos.org/" 2021-04-04T07:04:14.746 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-04-04T07:04:14.746 INFO:teuthology.orchestra.run.smithi097.stdout: 2021-04-04T07:04:14.747 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-04-04T07:04:14.747 INFO:teuthology.orchestra.run.smithi097.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-04-04T07:04:14.747 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-04-04T07:04:14.748 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-04-04T07:04:14.748 INFO:teuthology.orchestra.run.smithi097.stdout: 2021-04-04T07:04:14.749 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2021-04-04T07:04:14.852 INFO:teuthology.dispatcher.supervisor:Running job 6019052 2021-04-04T07:04:14.853 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-04-04_04:17:02-fs-pacific-distro-basic-smithi/6019052 --name teuthology-2021-04-04_04:17:02-fs-pacific-distro-basic-smithi --description fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} -- /home/teuthworker/archive/teuthology-2021-04-04_04:17:02-fs-pacific-distro-basic-smithi/6019052/orig.config.yaml 2021-04-04T07:04:14.858 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-04-04_04:17:02-fs-pacific-distro-basic-smithi/6019052 2021-04-04T07:04:14.859 INFO:teuthology.dispatcher.supervisor:Job PID: 13625 2021-04-04T07:04:14.860 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-04-04T19:04:31.576 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-04-04T19:04:31.803 INFO:teuthology.kill:Killing Pids: {13625} 2021-04-04T19:04:33.332 INFO:teuthology.kill:Nuking machines: ['smithi096', 'smithi097'] 2021-04-04T19:04:34.335 INFO:teuthology.kill:2021-04-04 19:04:34,334.334 INFO:teuthology.nuke:targets: 2021-04-04T19:04:34.336 INFO:teuthology.kill: smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC969O2G4lyA7i6Q+4Su9Y+gij0y8ijAUeKWF5ozToaeA6+NYFtyxEUZ7FxwPuDxiNKyikKAyTd2PQIIxfgbWVociqBQKdLsVDCYrNGjKLLE83w4RGOfHtSf6KOyCrqOP/47FpHchwXQ4u8CTorRW2b+xg0DSmnMhxnqSiBFFV573Iy5g1PrLYmkyVIdu2p5z2sH38h+WvWGbmB0/W6nDWukHtOsyVTahN10TrZvFiLOlVAejqDyM2vlyheHmWdyyFPGCVRwv4H4BwkQbDIQOeGb12Tv50VsOH1PcDL5xo0EZuLaONqhnu2tjYKclaDYTQVzZj2yu82oqD+rdEGh6oRQgrMQ5vlC+XT8JElYJiKZxtkZXzVjU5EEmiBsFPvgKQYcMI/2H9az7kA/mtUNdlXdCGMR5Pj183Sl0HL5uF30rOeLPOcNUKOF6nOxB+79CojmL2kUtab8z5myCVQWB7WZXLXzqvxn/aYrIPTTTV8bEUkGte1mBlFE8JZx1Kfqrc= 2021-04-04T19:04:34.336 INFO:teuthology.kill: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXhrT/jqPzZb/TkTf29wd0Fuk2fBdYKVIKXxGZAU83GEybE57vKFq5/vYOeEQNFovOx3mDXBsikon3dhi00mRkoSQPJTPzdPTpthJwSjbdyX4gPkXzJu3ngWWsLueBpVunYMAfDuxYYAV2c7PKz8Y2oQimOj7qd4OibBKjUkxiO87kSkA5nXMxFBwE/tOYlZW5Aoq/mnyUiMNUkBbYvwz3s9jTSx7mhUAZSy0vb9+36RKXJSTRP0Ck6nWBFzu35A98kS4q7YqJ6275FsFCOFdd6h32JXdMJ3JJTRg7dPD0D0l2VWVDh/ZMe4Z9Y4j2R5D3Jxu8C7JSFBglEY/xBLWKNhtujMRyu83+XjbMbz8o/abY/wGCiJ4aGU1SrZ2gpzKC+lvYsr82TZg7HtDBRZFbFq5V2qx/7vrKhHHOUs6RSryHzDvIi3Rba9qgcBaJTvmmV9I5Z2i9siU6rOMwwsZr/gIK2PS+PfDz3Iyqqr3dOIQcRxnYRCWgwcT+nDNdpec= 2021-04-04T19:04:34.336 INFO:teuthology.kill:2021-04-04 19:04:34,335.335 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-04T19:04:34.337 INFO:teuthology.kill:2021-04-04 19:04:34,337.337 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-04T19:04:34.412 INFO:teuthology.kill:2021-04-04 19:04:34,411.411 INFO:teuthology.orchestra.console:Power off smithi096 2021-04-04T19:04:34.427 INFO:teuthology.kill:2021-04-04 19:04:34,427.427 INFO:teuthology.orchestra.console:Power off smithi097 2021-04-04T19:04:42.801 INFO:teuthology.kill:2021-04-04 19:04:42,801.801 INFO:teuthology.orchestra.console:Power off for smithi096 completed 2021-04-04T19:04:46.920 INFO:teuthology.kill:2021-04-04 19:04:46,919.919 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-04-04T19:04:47.165 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-04-04T19:04:47.166 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2021-04-04T19:04:47.166 INFO:teuthology.misc:Compressing logs... 2021-04-04T19:04:47.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-04-04T19:05:21.755 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-04-04T19:05:21.827 INFO:teuthology.kill:No teuthology processes running 2021-04-04T19:05:23.603 INFO:teuthology.kill:Nuking machines: ['smithi096', 'smithi097'] 2021-04-04T19:05:24.556 INFO:teuthology.kill:2021-04-04 19:05:24,556.556 INFO:teuthology.nuke:targets: 2021-04-04T19:05:24.557 INFO:teuthology.kill: smithi096.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC969O2G4lyA7i6Q+4Su9Y+gij0y8ijAUeKWF5ozToaeA6+NYFtyxEUZ7FxwPuDxiNKyikKAyTd2PQIIxfgbWVociqBQKdLsVDCYrNGjKLLE83w4RGOfHtSf6KOyCrqOP/47FpHchwXQ4u8CTorRW2b+xg0DSmnMhxnqSiBFFV573Iy5g1PrLYmkyVIdu2p5z2sH38h+WvWGbmB0/W6nDWukHtOsyVTahN10TrZvFiLOlVAejqDyM2vlyheHmWdyyFPGCVRwv4H4BwkQbDIQOeGb12Tv50VsOH1PcDL5xo0EZuLaONqhnu2tjYKclaDYTQVzZj2yu82oqD+rdEGh6oRQgrMQ5vlC+XT8JElYJiKZxtkZXzVjU5EEmiBsFPvgKQYcMI/2H9az7kA/mtUNdlXdCGMR5Pj183Sl0HL5uF30rOeLPOcNUKOF6nOxB+79CojmL2kUtab8z5myCVQWB7WZXLXzqvxn/aYrIPTTTV8bEUkGte1mBlFE8JZx1Kfqrc= 2021-04-04T19:05:24.557 INFO:teuthology.kill: smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXhrT/jqPzZb/TkTf29wd0Fuk2fBdYKVIKXxGZAU83GEybE57vKFq5/vYOeEQNFovOx3mDXBsikon3dhi00mRkoSQPJTPzdPTpthJwSjbdyX4gPkXzJu3ngWWsLueBpVunYMAfDuxYYAV2c7PKz8Y2oQimOj7qd4OibBKjUkxiO87kSkA5nXMxFBwE/tOYlZW5Aoq/mnyUiMNUkBbYvwz3s9jTSx7mhUAZSy0vb9+36RKXJSTRP0Ck6nWBFzu35A98kS4q7YqJ6275FsFCOFdd6h32JXdMJ3JJTRg7dPD0D0l2VWVDh/ZMe4Z9Y4j2R5D3Jxu8C7JSFBglEY/xBLWKNhtujMRyu83+XjbMbz8o/abY/wGCiJ4aGU1SrZ2gpzKC+lvYsr82TZg7HtDBRZFbFq5V2qx/7vrKhHHOUs6RSryHzDvIi3Rba9qgcBaJTvmmV9I5Z2i9siU6rOMwwsZr/gIK2PS+PfDz3Iyqqr3dOIQcRxnYRCWgwcT+nDNdpec= 2021-04-04T19:05:24.579 INFO:teuthology.kill:2021-04-04 19:05:24,578.578 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-04T19:05:24.591 INFO:teuthology.kill:2021-04-04 19:05:24,591.591 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-04-04T19:05:24.658 INFO:teuthology.kill:2021-04-04 19:05:24,658.658 INFO:teuthology.orchestra.console:Power off smithi097 2021-04-04T19:05:24.676 INFO:teuthology.kill:2021-04-04 19:05:24,675.675 INFO:teuthology.orchestra.console:Power off smithi096 2021-04-04T19:05:24.841 INFO:teuthology.kill:2021-04-04 19:05:24,841.841 INFO:teuthology.orchestra.console:Power off for smithi097 completed 2021-04-04T19:05:24.851 INFO:teuthology.kill:2021-04-04 19:05:24,851.851 INFO:teuthology.orchestra.console:Power off for smithi096 completed 2021-04-04T19:05:55.191 INFO:teuthology.kill:2021-04-04 19:05:55,190.190 INFO:teuthology.lock.ops:unlocked smithi097.front.sepia.ceph.com 2021-04-04T19:05:55.209 INFO:teuthology.kill:2021-04-04 19:05:55,208.208 INFO:teuthology.lock.ops:unlocked smithi096.front.sepia.ceph.com 2021-04-04T19:07:55.837 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15