2021-08-11T19:58:07.511 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-11T19:58:07.619 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi049.front.sepia.ceph.com' reimaging 2021-08-11T19:58:07.620 INFO:teuthology.lock.ops:Updating [ubuntu@smithi049.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T19:58:07.620 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2021-08-11T19:58:07.636 INFO:teuthology.lock.ops:Node 'ubuntu@smithi049.front.sepia.ceph.com' reimaging is complete 2021-08-11T19:58:07.637 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi093.front.sepia.ceph.com' reimaging 2021-08-11T19:58:07.637 INFO:teuthology.lock.ops:Updating [ubuntu@smithi093.front.sepia.ceph.com]: reset os type and version on server 2021-08-11T19:58:07.638 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server 2021-08-11T19:58:07.652 INFO:teuthology.lock.ops:Node 'ubuntu@smithi093.front.sepia.ceph.com' reimaging is complete 2021-08-11T19:58:07.854 INFO:teuthology.provision.fog.smithi049:Scheduling deploy of centos 8.stream 2021-08-11T19:58:07.863 INFO:teuthology.provision.fog.smithi093:Scheduling deploy of centos 8.stream 2021-08-11T19:58:08.200 INFO:teuthology.orchestra.console:Power off smithi093 2021-08-11T19:58:08.213 INFO:teuthology.orchestra.console:Power off smithi049 2021-08-11T19:58:08.342 INFO:teuthology.orchestra.console:Power off for smithi093 completed 2021-08-11T19:58:08.354 INFO:teuthology.orchestra.console:Power off for smithi049 completed 2021-08-11T19:58:08.444 INFO:teuthology.orchestra.console:Power on smithi093 2021-08-11T19:58:08.455 INFO:teuthology.orchestra.console:Power on smithi049 2021-08-11T19:58:16.812 INFO:teuthology.orchestra.console:Power on for smithi093 completed 2021-08-11T19:58:16.818 INFO:teuthology.orchestra.console:Power on for smithi049 completed 2021-08-11T19:58:16.914 INFO:teuthology.provision.fog.smithi093:Waiting for deploy to finish 2021-08-11T19:58:16.919 INFO:teuthology.provision.fog.smithi049:Waiting for deploy to finish 2021-08-11T20:03:05.937 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T20:03:05.960 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-11T20:03:06.213 INFO:teuthology.provision.fog.smithi049:Node is ready 2021-08-11T20:03:06.230 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049.front.sepia.ceph.com 2021-08-11T20:03:06.248 INFO:teuthology.provision.fog.smithi093:Node is ready 2021-08-11T20:03:06.266 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093.front.sepia.ceph.com 2021-08-11T20:03:06.288 INFO:teuthology.orchestra.run.smithi049.stdout:172.21.15.49 smithi049.front.sepia.ceph.com smithi049 2021-08-11T20:03:06.324 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.15.93 smithi093.front.sepia.ceph.com smithi093 2021-08-11T20:03:06.639 INFO:teuthology.provision.fog.smithi049:Deploy complete! 2021-08-11T20:03:06.659 INFO:teuthology.provision.fog.smithi093:Deploy complete! 2021-08-11T20:03:06.817 INFO:teuthology.lock.ops:Checking smithi049.front.sepia.ceph.com 2021-08-11T20:03:06.818 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T20:03:06.836 INFO:teuthology.lock.ops:Checking smithi093.front.sepia.ceph.com 2021-08-11T20:03:06.836 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-11T20:03:06.851 INFO:teuthology.lock.ops:Updating [smithi049.front.sepia.ceph.com]: set os type and version on server 2021-08-11T20:03:06.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T20:03:07.078 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2021-08-11T20:03:07.133 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2021-08-11T20:03:07.134 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="8" 2021-08-11T20:03:07.134 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2021-08-11T20:03:07.134 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2021-08-11T20:03:07.135 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="8" 2021-08-11T20:03:07.135 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el8" 2021-08-11T20:03:07.135 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 8" 2021-08-11T20:03:07.135 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2021-08-11T20:03:07.136 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-11T20:03:07.136 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2021-08-11T20:03:07.136 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-08-11T20:03:07.136 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-08-11T20:03:07.136 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-08-11T20:03:07.137 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2021-08-11T20:03:07.153 INFO:teuthology.lock.ops:Updating [smithi093.front.sepia.ceph.com]: set os type and version on server 2021-08-11T20:03:07.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-11T20:03:07.380 INFO:teuthology.orchestra.run.smithi093.stdout:x86_64 2021-08-11T20:03:07.435 INFO:teuthology.orchestra.run.smithi093.stdout:NAME="CentOS Stream" 2021-08-11T20:03:07.436 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION="8" 2021-08-11T20:03:07.436 INFO:teuthology.orchestra.run.smithi093.stdout:ID="centos" 2021-08-11T20:03:07.436 INFO:teuthology.orchestra.run.smithi093.stdout:ID_LIKE="rhel fedora" 2021-08-11T20:03:07.437 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_ID="8" 2021-08-11T20:03:07.437 INFO:teuthology.orchestra.run.smithi093.stdout:PLATFORM_ID="platform:el8" 2021-08-11T20:03:07.437 INFO:teuthology.orchestra.run.smithi093.stdout:PRETTY_NAME="CentOS Stream 8" 2021-08-11T20:03:07.437 INFO:teuthology.orchestra.run.smithi093.stdout:ANSI_COLOR="0;31" 2021-08-11T20:03:07.438 INFO:teuthology.orchestra.run.smithi093.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-11T20:03:07.438 INFO:teuthology.orchestra.run.smithi093.stdout:HOME_URL="https://centos.org/" 2021-08-11T20:03:07.438 INFO:teuthology.orchestra.run.smithi093.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-08-11T20:03:07.438 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-08-11T20:03:07.438 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-08-11T20:03:07.439 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server 2021-08-11T20:03:07.519 INFO:teuthology.dispatcher.supervisor:Running job 6335135 2021-08-11T20:03:07.519 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-08-11_18:55:50-rados:thrash-wip-ronenf-unique-scrub-distro-basic-smithi/6335135 --name rfriedma-2021-08-11_18:55:50-rados:thrash-wip-ronenf-unique-scrub-distro-basic-smithi --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/small-objects-balanced} -- /home/teuthworker/archive/rfriedma-2021-08-11_18:55:50-rados:thrash-wip-ronenf-unique-scrub-distro-basic-smithi/6335135/orig.config.yaml 2021-08-11T20:03:07.523 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-08-11_18:55:50-rados:thrash-wip-ronenf-unique-scrub-distro-basic-smithi/6335135 2021-08-11T20:03:07.523 INFO:teuthology.dispatcher.supervisor:Job PID: 3236 2021-08-11T20:03:07.524 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-12T08:03:59.907 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-12T08:04:00.125 INFO:teuthology.kill:Killing Pids: {3236} 2021-08-12T08:04:01.281 INFO:teuthology.kill:Nuking machines: ['smithi049', 'smithi093'] 2021-08-12T08:04:02.211 INFO:teuthology.kill:2021-08-12 08:04:02,211.211 INFO:teuthology.nuke:targets: 2021-08-12T08:04:02.212 INFO:teuthology.kill: smithi049.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1wSV/s6w4PBg4P6ZoCAh6Acu8JaSGggjVjvFUvYmm23ZzF8HzfPeZr+hgMoT5Lw0BzQdSo/C6bW6e8X6JuB0fbx5ZSS2462Dvu02/VQxXJSiC1mQJEWEjXpY/c46MCoZpCZ7iL1/P496uGVVtYhH3l708T25+BZBL7bUZcOQTKNUgPE1WlVssI14M+se/MDagoopREzvguhvg7kW7p9NhdE6iAJT0PIICDQJ/rP5K7nzf6+c2dKUDZEGhVjrTRan99zmbUTNElvMFTSnZNqBeDw0kYOqC7STsFS3DiU97+57nPU7KO2T67sfeOypSqEvdRkKgzmH0AvBxPUVNHn49jFtNISEqxwN/j6M7oKzOKNpdXkga8e09BB9WkPRMOga9Hl7y4O57Gn+FZeEeliqEucfVpExUNNLUPoR4Psy8zhxi+jxmhouVXcNWBplWqy4aAbI0jQ99nh2hJN17Dgwadl+nVH9usx2oZs8DT8f/sfHcS49KOTk81lfdLUREFys= 2021-08-12T08:04:02.212 INFO:teuthology.kill: smithi093.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCisthwVQ+yo5tIyJdFbjKnRWgimIGOVTBQ+YBgehKB5IReYnakpFPuUW+r26fkO9f8ZfZ5YvkfMo52wfd5a7YEt193v9sP1cofoRN7u6nsjrNoAYXd8XSq2IiwPdgLg6M2mM50iiLv08ZH6a29V44vrLAHccXdTlFk8+NGzqw3FlK0SpoDG3mO9+vnZM4RoY9tggQhpnBjc8WdkOCIZNdIP8zzuZ2YXXkbWBboR+UbEQ4M6uJq3DIVDaOQitcG/16RQLA1NenPC9fzvcRhM2U0x2k+A5oeqm5h5v465TYDmwfq9xXwVYfeOJqa5ltxkaX3YHg0SlKf0d5JyYs6gNTIWVuH3yAolLbgNDPWa0E07LEPDwr68KbnNYf2u3M7aC6MQSGl2t4trERmfB9n9nRqG0eYIbYJL386adcSaDEzbRPoD0j5PgkhbGBD/NFNBB8/A63Qx0CSadBWH7PadQd2bhrM+c8J0aBNU4kD8KjvIAt7GXX2qmJl0dxCMvPQ+iU= 2021-08-12T08:04:02.212 INFO:teuthology.kill:2021-08-12 08:04:02,211.211 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T08:04:02.214 INFO:teuthology.kill:2021-08-12 08:04:02,214.214 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T08:04:02.287 INFO:teuthology.kill:2021-08-12 08:04:02,287.287 INFO:teuthology.orchestra.console:Power off smithi049 2021-08-12T08:04:02.303 INFO:teuthology.kill:2021-08-12 08:04:02,302.302 INFO:teuthology.orchestra.console:Power off smithi093 2021-08-12T08:04:10.660 INFO:teuthology.kill:2021-08-12 08:04:10,660.660 INFO:teuthology.orchestra.console:Power off for smithi049 completed 2021-08-12T08:04:14.780 INFO:teuthology.kill:2021-08-12 08:04:14,780.780 INFO:teuthology.orchestra.console:Power off for smithi093 completed 2021-08-12T08:04:15.041 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-08-12T08:04:15.042 INFO:teuthology.task.internal:roles: ubuntu@smithi093.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-08-12T08:04:15.042 INFO:teuthology.misc:Compressing logs... 2021-08-12T08:04:15.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-12T08:04:49.739 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-12T08:04:49.803 INFO:teuthology.kill:No teuthology processes running 2021-08-12T08:04:50.991 INFO:teuthology.kill:Nuking machines: ['smithi049', 'smithi093'] 2021-08-12T08:04:51.984 INFO:teuthology.kill:2021-08-12 08:04:51,984.984 INFO:teuthology.nuke:targets: 2021-08-12T08:04:51.985 INFO:teuthology.kill: smithi049.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1wSV/s6w4PBg4P6ZoCAh6Acu8JaSGggjVjvFUvYmm23ZzF8HzfPeZr+hgMoT5Lw0BzQdSo/C6bW6e8X6JuB0fbx5ZSS2462Dvu02/VQxXJSiC1mQJEWEjXpY/c46MCoZpCZ7iL1/P496uGVVtYhH3l708T25+BZBL7bUZcOQTKNUgPE1WlVssI14M+se/MDagoopREzvguhvg7kW7p9NhdE6iAJT0PIICDQJ/rP5K7nzf6+c2dKUDZEGhVjrTRan99zmbUTNElvMFTSnZNqBeDw0kYOqC7STsFS3DiU97+57nPU7KO2T67sfeOypSqEvdRkKgzmH0AvBxPUVNHn49jFtNISEqxwN/j6M7oKzOKNpdXkga8e09BB9WkPRMOga9Hl7y4O57Gn+FZeEeliqEucfVpExUNNLUPoR4Psy8zhxi+jxmhouVXcNWBplWqy4aAbI0jQ99nh2hJN17Dgwadl+nVH9usx2oZs8DT8f/sfHcS49KOTk81lfdLUREFys= 2021-08-12T08:04:51.986 INFO:teuthology.kill: smithi093.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCisthwVQ+yo5tIyJdFbjKnRWgimIGOVTBQ+YBgehKB5IReYnakpFPuUW+r26fkO9f8ZfZ5YvkfMo52wfd5a7YEt193v9sP1cofoRN7u6nsjrNoAYXd8XSq2IiwPdgLg6M2mM50iiLv08ZH6a29V44vrLAHccXdTlFk8+NGzqw3FlK0SpoDG3mO9+vnZM4RoY9tggQhpnBjc8WdkOCIZNdIP8zzuZ2YXXkbWBboR+UbEQ4M6uJq3DIVDaOQitcG/16RQLA1NenPC9fzvcRhM2U0x2k+A5oeqm5h5v465TYDmwfq9xXwVYfeOJqa5ltxkaX3YHg0SlKf0d5JyYs6gNTIWVuH3yAolLbgNDPWa0E07LEPDwr68KbnNYf2u3M7aC6MQSGl2t4trERmfB9n9nRqG0eYIbYJL386adcSaDEzbRPoD0j5PgkhbGBD/NFNBB8/A63Qx0CSadBWH7PadQd2bhrM+c8J0aBNU4kD8KjvIAt7GXX2qmJl0dxCMvPQ+iU= 2021-08-12T08:04:52.005 INFO:teuthology.kill:2021-08-12 08:04:52,005.005 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T08:04:52.009 INFO:teuthology.kill:2021-08-12 08:04:52,009.009 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-12T08:04:52.073 INFO:teuthology.kill:2021-08-12 08:04:52,073.073 INFO:teuthology.orchestra.console:Power off smithi093 2021-08-12T08:04:52.089 INFO:teuthology.kill:2021-08-12 08:04:52,089.089 INFO:teuthology.orchestra.console:Power off smithi049 2021-08-12T08:04:52.222 INFO:teuthology.kill:2021-08-12 08:04:52,221.221 INFO:teuthology.orchestra.console:Power off for smithi093 completed 2021-08-12T08:04:52.234 INFO:teuthology.kill:2021-08-12 08:04:52,234.234 INFO:teuthology.orchestra.console:Power off for smithi049 completed 2021-08-12T08:04:52.361 INFO:teuthology.kill:2021-08-12 08:04:52,361.361 INFO:teuthology.lock.ops:unlocked smithi093.front.sepia.ceph.com 2021-08-12T08:04:52.373 INFO:teuthology.kill:2021-08-12 08:04:52,373.373 INFO:teuthology.lock.ops:unlocked smithi049.front.sepia.ceph.com 2021-08-12T08:06:52.661 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-12T08:06:52.756 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-12T08:06:52.757 INFO:teuthology.nuke:Checking targets against current locks 2021-08-12T08:06:52.840 INFO:teuthology.nuke:Not nuking smithi049.front.sepia.ceph.com because description doesn't match 2021-08-12T08:06:52.841 INFO:teuthology.nuke:Not nuking smithi093.front.sepia.ceph.com because description doesn't match