2021-08-29T19:16:34.728 INFO:root:teuthology version: 1.1.0-df69d6db
2021-08-29T19:16:34.865 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi120.front.sepia.ceph.com' reimaging
2021-08-29T19:16:34.866 INFO:teuthology.lock.ops:Updating [ubuntu@smithi120.front.sepia.ceph.com]: reset os type and version on server
2021-08-29T19:16:34.867 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server
2021-08-29T19:16:34.884 INFO:teuthology.lock.ops:Node 'ubuntu@smithi120.front.sepia.ceph.com' reimaging is complete
2021-08-29T19:16:34.884 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging
2021-08-29T19:16:34.885 INFO:teuthology.lock.ops:Updating [ubuntu@smithi165.front.sepia.ceph.com]: reset os type and version on server
2021-08-29T19:16:34.885 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-08-29T19:16:34.903 INFO:teuthology.lock.ops:Node 'ubuntu@smithi165.front.sepia.ceph.com' reimaging is complete
2021-08-29T19:16:35.090 INFO:teuthology.provision.fog.smithi165:Scheduling deploy of centos 8.stream
2021-08-29T19:16:35.097 INFO:teuthology.provision.fog.smithi120:Scheduling deploy of centos 8.stream
2021-08-29T19:16:35.492 INFO:teuthology.orchestra.console:Power off smithi165
2021-08-29T19:16:35.517 INFO:teuthology.orchestra.console:Power off smithi120
2021-08-29T19:16:43.908 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-08-29T19:16:43.913 INFO:teuthology.orchestra.console:Power off for smithi120 completed
2021-08-29T19:16:44.010 INFO:teuthology.orchestra.console:Power on smithi165
2021-08-29T19:16:44.017 INFO:teuthology.orchestra.console:Power on smithi120
2021-08-29T19:16:52.373 INFO:teuthology.orchestra.console:Power on for smithi165 completed
2021-08-29T19:16:52.378 INFO:teuthology.orchestra.console:Power on for smithi120 completed
2021-08-29T19:16:52.476 INFO:teuthology.provision.fog.smithi165:Waiting for deploy to finish
2021-08-29T19:16:52.480 INFO:teuthology.provision.fog.smithi120:Waiting for deploy to finish
2021-08-29T19:20:59.321 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-08-29T19:21:19.439 INFO:teuthology.provision.fog.smithi120:Node is ready
2021-08-29T19:21:19.459 INFO:teuthology.orchestra.run.smithi120.stdout:smithi120.front.sepia.ceph.com
2021-08-29T19:21:19.517 INFO:teuthology.orchestra.run.smithi120.stdout:172.21.15.120 smithi120.front.sepia.ceph.com smithi120
2021-08-29T19:21:19.920 INFO:teuthology.provision.fog.smithi120:Deploy complete!
2021-08-29T19:21:48.842 INFO:teuthology.orchestra.run:Running command with timeout 600
2021-08-29T19:21:49.133 INFO:teuthology.provision.fog.smithi165:Node is ready
2021-08-29T19:21:49.149 INFO:teuthology.orchestra.run.smithi165.stdout:smithi165.front.sepia.ceph.com
2021-08-29T19:21:49.206 INFO:teuthology.orchestra.run.smithi165.stdout:172.21.15.165 smithi165.front.sepia.ceph.com smithi165
2021-08-29T19:21:49.578 INFO:teuthology.provision.fog.smithi165:Deploy complete!
2021-08-29T19:21:49.730 INFO:teuthology.lock.ops:Checking smithi120.front.sepia.ceph.com
2021-08-29T19:21:49.731 INFO:teuthology.lock.ops:New key found. Updating...
2021-08-29T19:21:49.750 INFO:teuthology.lock.ops:Checking smithi165.front.sepia.ceph.com
2021-08-29T19:21:49.750 INFO:teuthology.lock.ops:New key found. Updating...
2021-08-29T19:21:49.764 INFO:teuthology.lock.ops:Updating [smithi120.front.sepia.ceph.com]: set os type and version on server
2021-08-29T19:21:49.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-08-29T19:21:50.003 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64
2021-08-29T19:21:50.060 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream"
2021-08-29T19:21:50.060 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="8"
2021-08-29T19:21:50.060 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos"
2021-08-29T19:21:50.061 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora"
2021-08-29T19:21:50.061 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="8"
2021-08-29T19:21:50.061 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el8"
2021-08-29T19:21:50.061 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 8"
2021-08-29T19:21:50.062 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31"
2021-08-29T19:21:50.062 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-08-29T19:21:50.062 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/"
2021-08-29T19:21:50.062 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-08-29T19:21:50.063 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-08-29T19:21:50.063 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-08-29T19:21:50.064 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server
2021-08-29T19:21:50.080 INFO:teuthology.lock.ops:Updating [smithi165.front.sepia.ceph.com]: set os type and version on server
2021-08-29T19:21:50.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-08-29T19:21:50.311 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64
2021-08-29T19:21:50.366 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream"
2021-08-29T19:21:50.367 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="8"
2021-08-29T19:21:50.367 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos"
2021-08-29T19:21:50.367 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora"
2021-08-29T19:21:50.368 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="8"
2021-08-29T19:21:50.368 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el8"
2021-08-29T19:21:50.368 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 8"
2021-08-29T19:21:50.368 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31"
2021-08-29T19:21:50.368 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2021-08-29T19:21:50.369 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/"
2021-08-29T19:21:50.369 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2021-08-29T19:21:50.369 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2021-08-29T19:21:50.369 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2021-08-29T19:21:50.370 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server
2021-08-29T19:21:50.479 INFO:teuthology.dispatcher.supervisor:Running job 6365927
2021-08-29T19:21:50.480 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-08-29_18:38:12-rados:thrash-wip-ronenf-scrub-token-osdlog-distro-basic-smithi/6365927 --name rfriedma-2021-08-29_18:38:12-rados:thrash-wip-ronenf-scrub-token-osdlog-distro-basic-smithi --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/small-objects} -- /home/teuthworker/archive/rfriedma-2021-08-29_18:38:12-rados:thrash-wip-ronenf-scrub-token-osdlog-distro-basic-smithi/6365927/orig.config.yaml
2021-08-29T19:21:50.482 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-08-29_18:38:12-rados:thrash-wip-ronenf-scrub-token-osdlog-distro-basic-smithi/6365927
2021-08-29T19:21:50.483 INFO:teuthology.dispatcher.supervisor:Job PID: 11242
2021-08-29T19:21:50.483 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2021-08-30T07:22:32.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2021-08-30T07:22:33.014 INFO:teuthology.kill:Killing Pids: {11242}
2021-08-30T07:22:34.128 INFO:teuthology.kill:Nuking machines: ['smithi120', 'smithi165']
2021-08-30T07:22:35.095 INFO:teuthology.kill:2021-08-30 07:22:35,095.095 INFO:teuthology.nuke:targets:
2021-08-30T07:22:35.096 INFO:teuthology.kill:  smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWjg9Z9F13iqIxjRgDRt0m4Mdt4LsJiT86ItahBAWCvKG7L3qY5ng2iGE9WS3Kf54x+x3M77a3WP7bw4id/f9H68VgDLJQdQ3mwSfyFMog/7U59wI5Mk3Xpnned8QHOwpa5kqUzVmBwk7jNLjWpgb/k43C19QyZ/TchwCEcYvGlPUdGfS/rg2EbZ43tuY23FJk22RZVtkmagvWn1HRl/FFiyKyUqd0Id3NT0IGwVASU0Zvqneixf6Wv8avr7wLtlLKptf58OG5JLay0PW4hTpt+RMwWczbHxiO76KhEd6ZjtfHE4RiiMRN+TTMiHWM24HpLiEhPV9bV6Jhg+7F2pB1QTdyShFs1CsRxWT0zDMr3cHs3KuX5BnloZvlTE1f9Igpx4swu4AzvlIwq6tnH++JiRNtLO1q65TFZH99f9lCInvX987cpJTB6FFCMV7mwkke9gfgxd6jiACTDqY1nDBwSsFDvpzWOV/ap1ooIGTn8B3DwtEZXXcF2ZStXkbXW+k=
2021-08-30T07:22:35.096 INFO:teuthology.kill:  smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBQc77JYi3wgrHGTeSk6CpwkZ62qNkPD7sxKyTfdHJ906Cb9fqek0NLTFlzoo4u/+EEMnArDwmCMVZutFu172hQ6sNwPwJlLRCMXev0/wTbkIDyu/0T/moGV0gIFYNT9ofC7AvTgmj8PgM2LhL+eXyM1vUwawI1eI4B2tPiCDnxdCpwXFofXGFOpUqq0Aw7v0cXH08GdYrzxsrLoZvjD6E26tpqgg7kXuX7r+EA50t8PHhfDym/+PMzWrMbcSVIeQP7tVHuRN41FksDnBT+N9iniF/v0yKP3c7fO/ICocdvychUDZyqFmoJqpsT89ddjN5sTA7f6u8KpgIKvfRVDqhWEAXvloiOQL9lt/1irSPnFBQP9LymGcYKSN9FWFJtqPzjH+hFBptjTG4JsCUvUvVn2/Zkq3jQ68UpdEPjc50U2+9tiHa0iSeNuGNCAEHvTWNTInIfo6bAdGl0kFZEbXFmGNwBiSI4vSQ44iGurxl1smiLbWGeN7jshrUIJ2cHfU=
2021-08-30T07:22:35.097 INFO:teuthology.kill:2021-08-30 07:22:35,095.095 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-08-30T07:22:35.098 INFO:teuthology.kill:2021-08-30 07:22:35,098.098 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-08-30T07:22:35.159 INFO:teuthology.kill:2021-08-30 07:22:35,158.158 INFO:teuthology.orchestra.console:Power off smithi165
2021-08-30T07:22:35.172 INFO:teuthology.kill:2021-08-30 07:22:35,172.172 INFO:teuthology.orchestra.console:Power off smithi120
2021-08-30T07:22:47.648 INFO:teuthology.kill:2021-08-30 07:22:47,647.647 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-08-30T07:22:47.652 INFO:teuthology.kill:2021-08-30 07:22:47,652.652 INFO:teuthology.orchestra.console:Power off for smithi120 completed
2021-08-30T07:22:47.899 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2021-08-30T07:22:47.900 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2021-08-30T07:22:47.900 INFO:teuthology.misc:Compressing logs...
2021-08-30T07:22:47.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host
2021-08-30T07:23:47.926 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog
    teuth_config.archive_base, job_config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/misc.py", line 1378, in compress_logs
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/cluster.py", line 64, in <listcomp>
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_df69d6db9d68f5492f47c1dee57bca3b0a385796/teuthology/orchestra/run.py", line 434, in run
    transport = client.get_transport()
AttributeError: 'NoneType' object has no attribute 'get_transport'
2021-08-30T07:23:47.992 INFO:teuthology.kill:No teuthology processes running
2021-08-30T07:23:49.113 INFO:teuthology.kill:Nuking machines: ['smithi120', 'smithi165']
2021-08-30T07:23:50.066 INFO:teuthology.kill:2021-08-30 07:23:50,065.065 INFO:teuthology.nuke:targets:
2021-08-30T07:23:50.067 INFO:teuthology.kill:  smithi120.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWjg9Z9F13iqIxjRgDRt0m4Mdt4LsJiT86ItahBAWCvKG7L3qY5ng2iGE9WS3Kf54x+x3M77a3WP7bw4id/f9H68VgDLJQdQ3mwSfyFMog/7U59wI5Mk3Xpnned8QHOwpa5kqUzVmBwk7jNLjWpgb/k43C19QyZ/TchwCEcYvGlPUdGfS/rg2EbZ43tuY23FJk22RZVtkmagvWn1HRl/FFiyKyUqd0Id3NT0IGwVASU0Zvqneixf6Wv8avr7wLtlLKptf58OG5JLay0PW4hTpt+RMwWczbHxiO76KhEd6ZjtfHE4RiiMRN+TTMiHWM24HpLiEhPV9bV6Jhg+7F2pB1QTdyShFs1CsRxWT0zDMr3cHs3KuX5BnloZvlTE1f9Igpx4swu4AzvlIwq6tnH++JiRNtLO1q65TFZH99f9lCInvX987cpJTB6FFCMV7mwkke9gfgxd6jiACTDqY1nDBwSsFDvpzWOV/ap1ooIGTn8B3DwtEZXXcF2ZStXkbXW+k=
2021-08-30T07:23:50.067 INFO:teuthology.kill:  smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDBQc77JYi3wgrHGTeSk6CpwkZ62qNkPD7sxKyTfdHJ906Cb9fqek0NLTFlzoo4u/+EEMnArDwmCMVZutFu172hQ6sNwPwJlLRCMXev0/wTbkIDyu/0T/moGV0gIFYNT9ofC7AvTgmj8PgM2LhL+eXyM1vUwawI1eI4B2tPiCDnxdCpwXFofXGFOpUqq0Aw7v0cXH08GdYrzxsrLoZvjD6E26tpqgg7kXuX7r+EA50t8PHhfDym/+PMzWrMbcSVIeQP7tVHuRN41FksDnBT+N9iniF/v0yKP3c7fO/ICocdvychUDZyqFmoJqpsT89ddjN5sTA7f6u8KpgIKvfRVDqhWEAXvloiOQL9lt/1irSPnFBQP9LymGcYKSN9FWFJtqPzjH+hFBptjTG4JsCUvUvVn2/Zkq3jQ68UpdEPjc50U2+9tiHa0iSeNuGNCAEHvTWNTInIfo6bAdGl0kFZEbXFmGNwBiSI4vSQ44iGurxl1smiLbWGeN7jshrUIJ2cHfU=
2021-08-30T07:23:50.087 INFO:teuthology.kill:2021-08-30 07:23:50,087.087 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-08-30T07:23:50.090 INFO:teuthology.kill:2021-08-30 07:23:50,090.090 INFO:teuthology.task.internal.check_lock:Checking locks...
2021-08-30T07:23:50.144 INFO:teuthology.kill:2021-08-30 07:23:50,144.144 INFO:teuthology.orchestra.console:Power off smithi120
2021-08-30T07:23:50.161 INFO:teuthology.kill:2021-08-30 07:23:50,160.160 INFO:teuthology.orchestra.console:Power off smithi165
2021-08-30T07:23:50.292 INFO:teuthology.kill:2021-08-30 07:23:50,291.291 INFO:teuthology.orchestra.console:Power off for smithi120 completed
2021-08-30T07:23:50.297 INFO:teuthology.kill:2021-08-30 07:23:50,297.297 INFO:teuthology.orchestra.console:Power off for smithi165 completed
2021-08-30T07:23:50.433 INFO:teuthology.kill:2021-08-30 07:23:50,433.433 INFO:teuthology.lock.ops:unlocked smithi120.front.sepia.ceph.com
2021-08-30T07:23:50.436 INFO:teuthology.kill:2021-08-30 07:23:50,436.436 INFO:teuthology.lock.ops:unlocked smithi165.front.sepia.ceph.com
2021-08-30T07:25:50.618 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2021-08-30T07:25:50.711 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2021-08-30T07:25:50.712 INFO:teuthology.nuke:Checking targets against current locks
2021-08-30T07:25:50.808 INFO:teuthology.nuke:Not nuking smithi165.front.sepia.ceph.com because description doesn't match
2021-08-30T07:25:50.808 INFO:teuthology.nuke:Not nuking smithi120.front.sepia.ceph.com because description doesn't match