2021-09-15T15:52:10.992 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-15T15:52:11.102 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging 2021-09-15T15:52:11.103 INFO:teuthology.lock.ops:Updating [ubuntu@smithi085.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T15:52:11.104 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-09-15T15:52:11.120 INFO:teuthology.lock.ops:Node 'ubuntu@smithi085.front.sepia.ceph.com' reimaging is complete 2021-09-15T15:52:11.120 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging 2021-09-15T15:52:11.121 INFO:teuthology.lock.ops:Updating [ubuntu@smithi090.front.sepia.ceph.com]: reset os type and version on server 2021-09-15T15:52:11.121 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2021-09-15T15:52:11.138 INFO:teuthology.lock.ops:Node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging is complete 2021-09-15T15:52:11.358 INFO:teuthology.provision.fog.smithi085:Scheduling deploy of centos 8.stream 2021-09-15T15:52:11.380 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of centos 8.stream 2021-09-15T15:52:11.768 INFO:teuthology.orchestra.console:Power off smithi085 2021-09-15T15:52:11.857 INFO:teuthology.orchestra.console:Power off smithi090 2021-09-15T15:52:20.146 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-09-15T15:52:20.222 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2021-09-15T15:52:20.250 INFO:teuthology.orchestra.console:Power on smithi085 2021-09-15T15:52:20.323 INFO:teuthology.orchestra.console:Power on smithi090 2021-09-15T15:52:28.683 INFO:teuthology.orchestra.console:Power on for smithi085 completed 2021-09-15T15:52:28.698 INFO:teuthology.orchestra.console:Power on for smithi090 completed 2021-09-15T15:52:28.785 INFO:teuthology.provision.fog.smithi085:Waiting for deploy to finish 2021-09-15T15:52:28.800 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish 2021-09-15T15:56:41.298 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T15:57:02.057 INFO:teuthology.provision.fog.smithi085:Node is ready 2021-09-15T15:57:02.077 INFO:teuthology.orchestra.run.smithi085.stdout:smithi085.front.sepia.ceph.com 2021-09-15T15:57:02.136 INFO:teuthology.orchestra.run.smithi085.stdout:172.21.15.85 smithi085.front.sepia.ceph.com smithi085 2021-09-15T15:57:02.491 INFO:teuthology.provision.fog.smithi085:Deploy complete! 2021-09-15T15:57:40.159 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-09-15T15:57:40.518 INFO:teuthology.provision.fog.smithi090:Node is ready 2021-09-15T15:57:40.536 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com 2021-09-15T15:57:40.592 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090 2021-09-15T15:57:40.941 INFO:teuthology.provision.fog.smithi090:Deploy complete! 2021-09-15T15:57:41.079 INFO:teuthology.lock.ops:Checking smithi085.front.sepia.ceph.com 2021-09-15T15:57:41.080 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-15T15:57:41.134 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com 2021-09-15T15:57:41.134 INFO:teuthology.lock.ops:New key found. Updating... 2021-09-15T15:57:41.161 INFO:teuthology.lock.ops:Updating [smithi085.front.sepia.ceph.com]: set os type and version on server 2021-09-15T15:57:41.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-15T15:57:41.400 INFO:teuthology.orchestra.run.smithi085.stdout:x86_64 2021-09-15T15:57:41.456 INFO:teuthology.orchestra.run.smithi085.stdout:NAME="CentOS Stream" 2021-09-15T15:57:41.456 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION="8" 2021-09-15T15:57:41.457 INFO:teuthology.orchestra.run.smithi085.stdout:ID="centos" 2021-09-15T15:57:41.457 INFO:teuthology.orchestra.run.smithi085.stdout:ID_LIKE="rhel fedora" 2021-09-15T15:57:41.457 INFO:teuthology.orchestra.run.smithi085.stdout:VERSION_ID="8" 2021-09-15T15:57:41.457 INFO:teuthology.orchestra.run.smithi085.stdout:PLATFORM_ID="platform:el8" 2021-09-15T15:57:41.458 INFO:teuthology.orchestra.run.smithi085.stdout:PRETTY_NAME="CentOS Stream 8" 2021-09-15T15:57:41.458 INFO:teuthology.orchestra.run.smithi085.stdout:ANSI_COLOR="0;31" 2021-09-15T15:57:41.458 INFO:teuthology.orchestra.run.smithi085.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-15T15:57:41.458 INFO:teuthology.orchestra.run.smithi085.stdout:HOME_URL="https://centos.org/" 2021-09-15T15:57:41.459 INFO:teuthology.orchestra.run.smithi085.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-15T15:57:41.459 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-15T15:57:41.459 INFO:teuthology.orchestra.run.smithi085.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-09-15T15:57:41.460 INFO:teuthology.lock.ops:Updating smithi085.front.sepia.ceph.com on lock server 2021-09-15T15:57:41.475 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server 2021-09-15T15:57:41.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-15T15:57:41.699 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2021-09-15T15:57:41.755 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2021-09-15T15:57:41.755 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8" 2021-09-15T15:57:41.755 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2021-09-15T15:57:41.755 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2021-09-15T15:57:41.756 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8" 2021-09-15T15:57:41.756 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2021-09-15T15:57:41.756 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 8" 2021-09-15T15:57:41.757 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2021-09-15T15:57:41.757 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-09-15T15:57:41.757 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2021-09-15T15:57:41.757 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-09-15T15:57:41.758 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-09-15T15:57:41.758 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-09-15T15:57:41.759 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2021-09-15T15:57:41.842 INFO:teuthology.dispatcher.supervisor:Running job 6390938 2021-09-15T15:57:41.843 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-09-15_14:31:46-rados-wip-yuri4-testing-2021-09-14-1654-distro-basic-smithi/6390938 --name yuriw-2021-09-15_14:31:46-rados-wip-yuri4-testing-2021-09-14-1654-distro-basic-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8.stream} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} -- /home/teuthworker/archive/yuriw-2021-09-15_14:31:46-rados-wip-yuri4-testing-2021-09-14-1654-distro-basic-smithi/6390938/orig.config.yaml 2021-09-15T15:57:41.846 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-09-15_14:31:46-rados-wip-yuri4-testing-2021-09-14-1654-distro-basic-smithi/6390938 2021-09-15T15:57:41.847 INFO:teuthology.dispatcher.supervisor:Job PID: 5234 2021-09-15T15:57:41.848 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-09-16T03:58:33.167 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-09-16T03:58:33.350 INFO:teuthology.kill:Killing Pids: {5234} 2021-09-16T03:58:34.484 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi090'] 2021-09-16T03:58:35.444 INFO:teuthology.kill:2021-09-16 03:58:35,444.444 INFO:teuthology.nuke:targets: 2021-09-16T03:58:35.445 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD2uOuWQ5cB895RO9LIwOetdvYN+m0BgulnLsaoQPIsPVc7b8s8/PkKzzAVktiChrQ65e5YNCMOUdDSAtBZhH1AfqhsydSWKRTRDGWag79kc11LayWTs9v0vBaM4Y/8fTmNXfsYd5lBF7fnZFsGqfqgnb4SZDPH/pyg/oCt11Bqch3l15q8Ld/1vq0DelfmZHi60MtrANC17Gib/OT130ECY49sgqEeBbEpKSkzqOYmlWccfGF0i7Kpgs6JU1RG6Mz7thjaXVGsD4mdCR6LsvRVfc51PqKmv8VSPY9r4WZj6FB2AYNluI+O/GU4bzTe8psFgSH2IE965326c/HkiYnAYen3zlBXD+1n0B/RV/eXXVpfd8t+hMqHxH7BGxMdReZBLM36YciFDzQsXTyuPIXrMZsuHgw570BCnv3MuW7QfQnqnlkNHj3E+z7crLs20Ibgxc9YBWwMaR/WOyv3Rw3J/kCIvISIrZdySOVtpexfkUjJhUwz88TxAqhIxhnB77M= 2021-09-16T03:58:35.446 INFO:teuthology.kill: smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/G6YqINxlYiW37bEDPLn/C94ZZo4Efldli9prCGr7r45XGgeDOUNfDI+wgl30Kp60YxD2Uda7pue5o5Ut5eUbnKjpB8jJQmADIORjbTqA58lX6UURVOHRzj3OC1N47qFc2cecH0qK2tW0KeTsd5SecM/zL1Scu9Lvpn9EM2cUrw1vX6NWgqMw1+H6ue0sS8J8u1wMX/PLj5aFcO5UlJoDjp8rPcw2XNCmBkMsxlEVyEVLjL+ZYZw6hEoplTlxHRtuMQe9e0KMc5p32AYf5dcPmsJw6t1f26mp5UxkdqWjjAigHPe6ArCadmV1XkiBoa4xENZdaPx1A6d+VN0EZi7LSIgPvetNu4ni8/r/jRlxVFvmvXyt1dI9i/2lUNpG5SS7aSrWQ0QZhuibNVr/C+6RViliC5r6w0127+Jrh5HQ1Z2hYo8BTVEzpXA91HThI3QJe22IMQnVIuDNactt2L9f2/pqQhLJv+byZmDtyJ9afi4NScY9sqIqOSpG+9+TvMM= 2021-09-16T03:58:35.446 INFO:teuthology.kill:2021-09-16 03:58:35,444.444 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T03:58:35.447 INFO:teuthology.kill:2021-09-16 03:58:35,446.446 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T03:58:35.509 INFO:teuthology.kill:2021-09-16 03:58:35,508.508 INFO:teuthology.orchestra.console:Power off smithi085 2021-09-16T03:58:35.522 INFO:teuthology.kill:2021-09-16 03:58:35,521.521 INFO:teuthology.orchestra.console:Power off smithi090 2021-09-16T03:58:43.883 INFO:teuthology.kill:2021-09-16 03:58:43,882.882 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-09-16T03:58:43.887 INFO:teuthology.kill:2021-09-16 03:58:43,887.887 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2021-09-16T03:58:44.104 INFO:teuthology.task.internal:roles: ubuntu@smithi085.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-09-16T03:58:44.105 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-09-16T03:58:44.105 INFO:teuthology.misc:Compressing logs... 2021-09-16T03:58:44.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-09-16T03:59:44.146 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b135a4f82b02b966053efba0501e6a47c9d2d793/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-09-16T03:59:44.205 INFO:teuthology.kill:No teuthology processes running 2021-09-16T03:59:45.281 INFO:teuthology.kill:Nuking machines: ['smithi085', 'smithi090'] 2021-09-16T03:59:46.183 INFO:teuthology.kill:2021-09-16 03:59:46,182.182 INFO:teuthology.nuke:targets: 2021-09-16T03:59:46.184 INFO:teuthology.kill: smithi085.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD2uOuWQ5cB895RO9LIwOetdvYN+m0BgulnLsaoQPIsPVc7b8s8/PkKzzAVktiChrQ65e5YNCMOUdDSAtBZhH1AfqhsydSWKRTRDGWag79kc11LayWTs9v0vBaM4Y/8fTmNXfsYd5lBF7fnZFsGqfqgnb4SZDPH/pyg/oCt11Bqch3l15q8Ld/1vq0DelfmZHi60MtrANC17Gib/OT130ECY49sgqEeBbEpKSkzqOYmlWccfGF0i7Kpgs6JU1RG6Mz7thjaXVGsD4mdCR6LsvRVfc51PqKmv8VSPY9r4WZj6FB2AYNluI+O/GU4bzTe8psFgSH2IE965326c/HkiYnAYen3zlBXD+1n0B/RV/eXXVpfd8t+hMqHxH7BGxMdReZBLM36YciFDzQsXTyuPIXrMZsuHgw570BCnv3MuW7QfQnqnlkNHj3E+z7crLs20Ibgxc9YBWwMaR/WOyv3Rw3J/kCIvISIrZdySOVtpexfkUjJhUwz88TxAqhIxhnB77M= 2021-09-16T03:59:46.184 INFO:teuthology.kill: smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/G6YqINxlYiW37bEDPLn/C94ZZo4Efldli9prCGr7r45XGgeDOUNfDI+wgl30Kp60YxD2Uda7pue5o5Ut5eUbnKjpB8jJQmADIORjbTqA58lX6UURVOHRzj3OC1N47qFc2cecH0qK2tW0KeTsd5SecM/zL1Scu9Lvpn9EM2cUrw1vX6NWgqMw1+H6ue0sS8J8u1wMX/PLj5aFcO5UlJoDjp8rPcw2XNCmBkMsxlEVyEVLjL+ZYZw6hEoplTlxHRtuMQe9e0KMc5p32AYf5dcPmsJw6t1f26mp5UxkdqWjjAigHPe6ArCadmV1XkiBoa4xENZdaPx1A6d+VN0EZi7LSIgPvetNu4ni8/r/jRlxVFvmvXyt1dI9i/2lUNpG5SS7aSrWQ0QZhuibNVr/C+6RViliC5r6w0127+Jrh5HQ1Z2hYo8BTVEzpXA91HThI3QJe22IMQnVIuDNactt2L9f2/pqQhLJv+byZmDtyJ9afi4NScY9sqIqOSpG+9+TvMM= 2021-09-16T03:59:46.205 INFO:teuthology.kill:2021-09-16 03:59:46,204.204 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T03:59:46.208 INFO:teuthology.kill:2021-09-16 03:59:46,208.208 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-09-16T03:59:46.262 INFO:teuthology.kill:2021-09-16 03:59:46,261.261 INFO:teuthology.orchestra.console:Power off smithi085 2021-09-16T03:59:46.309 INFO:teuthology.kill:2021-09-16 03:59:46,308.308 INFO:teuthology.orchestra.console:Power off smithi090 2021-09-16T03:59:46.401 INFO:teuthology.kill:2021-09-16 03:59:46,400.400 INFO:teuthology.orchestra.console:Power off for smithi085 completed 2021-09-16T03:59:46.449 INFO:teuthology.kill:2021-09-16 03:59:46,449.449 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2021-09-16T03:59:46.542 INFO:teuthology.kill:2021-09-16 03:59:46,541.541 INFO:teuthology.lock.ops:unlocked smithi085.front.sepia.ceph.com 2021-09-16T03:59:46.584 INFO:teuthology.kill:2021-09-16 03:59:46,584.584 INFO:teuthology.lock.ops:unlocked smithi090.front.sepia.ceph.com 2021-09-16T04:01:47.876 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-09-16T04:01:47.966 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-09-16T04:01:47.967 INFO:teuthology.nuke:Checking targets against current locks 2021-09-16T04:01:48.054 INFO:teuthology.nuke:Not nuking smithi090.front.sepia.ceph.com because description doesn't match 2021-09-16T04:01:48.054 INFO:teuthology.nuke:Not nuking smithi085.front.sepia.ceph.com because description doesn't match