2021-08-17T16:27:26.027 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-17T16:27:26.127 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging 2021-08-17T16:27:26.128 INFO:teuthology.lock.ops:Updating [ubuntu@smithi084.front.sepia.ceph.com]: reset os type and version on server 2021-08-17T16:27:26.128 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-08-17T16:27:26.148 INFO:teuthology.lock.ops:Node 'ubuntu@smithi084.front.sepia.ceph.com' reimaging is complete 2021-08-17T16:27:26.149 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi205.front.sepia.ceph.com' reimaging 2021-08-17T16:27:26.149 INFO:teuthology.lock.ops:Updating [ubuntu@smithi205.front.sepia.ceph.com]: reset os type and version on server 2021-08-17T16:27:26.149 INFO:teuthology.lock.ops:Updating smithi205.front.sepia.ceph.com on lock server 2021-08-17T16:27:26.167 INFO:teuthology.lock.ops:Node 'ubuntu@smithi205.front.sepia.ceph.com' reimaging is complete 2021-08-17T16:27:26.359 INFO:teuthology.provision.fog.smithi084:Scheduling deploy of centos 8.stream 2021-08-17T16:27:26.369 INFO:teuthology.provision.fog.smithi205:Scheduling deploy of centos 8.stream 2021-08-17T16:27:26.731 INFO:teuthology.orchestra.console:Power off smithi084 2021-08-17T16:27:26.746 INFO:teuthology.orchestra.console:Power off smithi205 2021-08-17T16:27:26.874 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-08-17T16:27:26.885 INFO:teuthology.orchestra.console:Power off for smithi205 completed 2021-08-17T16:27:26.975 INFO:teuthology.orchestra.console:Power on smithi084 2021-08-17T16:27:26.986 INFO:teuthology.orchestra.console:Power on smithi205 2021-08-17T16:27:35.345 INFO:teuthology.orchestra.console:Power on for smithi084 completed 2021-08-17T16:27:35.350 INFO:teuthology.orchestra.console:Power on for smithi205 completed 2021-08-17T16:27:35.447 INFO:teuthology.provision.fog.smithi084:Waiting for deploy to finish 2021-08-17T16:27:35.451 INFO:teuthology.provision.fog.smithi205:Waiting for deploy to finish 2021-08-17T16:32:16.432 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-17T16:32:16.558 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-08-17T16:32:16.680 INFO:teuthology.provision.fog.smithi084:Node is ready 2021-08-17T16:32:16.698 INFO:teuthology.orchestra.run.smithi084.stdout:smithi084.front.sepia.ceph.com 2021-08-17T16:32:16.755 INFO:teuthology.orchestra.run.smithi084.stdout:172.21.15.84 smithi084.front.sepia.ceph.com smithi084 2021-08-17T16:32:16.832 INFO:teuthology.provision.fog.smithi205:Node is ready 2021-08-17T16:32:16.890 INFO:teuthology.orchestra.run.smithi205.stdout:smithi205.front.sepia.ceph.com 2021-08-17T16:32:16.947 INFO:teuthology.orchestra.run.smithi205.stdout:172.21.15.205 smithi205.front.sepia.ceph.com smithi205 2021-08-17T16:32:17.101 INFO:teuthology.provision.fog.smithi084:Deploy complete! 2021-08-17T16:32:17.280 INFO:teuthology.provision.fog.smithi205:Deploy complete! 2021-08-17T16:32:17.418 INFO:teuthology.lock.ops:Checking smithi084.front.sepia.ceph.com 2021-08-17T16:32:17.419 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-17T16:32:17.439 INFO:teuthology.lock.ops:Checking smithi205.front.sepia.ceph.com 2021-08-17T16:32:17.439 INFO:teuthology.lock.ops:New key found. Updating... 2021-08-17T16:32:17.456 INFO:teuthology.lock.ops:Updating [smithi084.front.sepia.ceph.com]: set os type and version on server 2021-08-17T16:32:17.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-17T16:32:17.680 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2021-08-17T16:32:17.737 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2021-08-17T16:32:17.737 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="8" 2021-08-17T16:32:17.738 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2021-08-17T16:32:17.738 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2021-08-17T16:32:17.738 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="8" 2021-08-17T16:32:17.738 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el8" 2021-08-17T16:32:17.739 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 8" 2021-08-17T16:32:17.739 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2021-08-17T16:32:17.739 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-17T16:32:17.739 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2021-08-17T16:32:17.740 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-08-17T16:32:17.740 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-08-17T16:32:17.740 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-08-17T16:32:17.742 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2021-08-17T16:32:17.762 INFO:teuthology.lock.ops:Updating [smithi205.front.sepia.ceph.com]: set os type and version on server 2021-08-17T16:32:17.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-17T16:32:17.989 INFO:teuthology.orchestra.run.smithi205.stdout:x86_64 2021-08-17T16:32:18.045 INFO:teuthology.orchestra.run.smithi205.stdout:NAME="CentOS Stream" 2021-08-17T16:32:18.045 INFO:teuthology.orchestra.run.smithi205.stdout:VERSION="8" 2021-08-17T16:32:18.045 INFO:teuthology.orchestra.run.smithi205.stdout:ID="centos" 2021-08-17T16:32:18.045 INFO:teuthology.orchestra.run.smithi205.stdout:ID_LIKE="rhel fedora" 2021-08-17T16:32:18.046 INFO:teuthology.orchestra.run.smithi205.stdout:VERSION_ID="8" 2021-08-17T16:32:18.046 INFO:teuthology.orchestra.run.smithi205.stdout:PLATFORM_ID="platform:el8" 2021-08-17T16:32:18.046 INFO:teuthology.orchestra.run.smithi205.stdout:PRETTY_NAME="CentOS Stream 8" 2021-08-17T16:32:18.046 INFO:teuthology.orchestra.run.smithi205.stdout:ANSI_COLOR="0;31" 2021-08-17T16:32:18.047 INFO:teuthology.orchestra.run.smithi205.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-17T16:32:18.047 INFO:teuthology.orchestra.run.smithi205.stdout:HOME_URL="https://centos.org/" 2021-08-17T16:32:18.047 INFO:teuthology.orchestra.run.smithi205.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-08-17T16:32:18.047 INFO:teuthology.orchestra.run.smithi205.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-08-17T16:32:18.048 INFO:teuthology.orchestra.run.smithi205.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-08-17T16:32:18.048 INFO:teuthology.lock.ops:Updating smithi205.front.sepia.ceph.com on lock server 2021-08-17T16:32:18.130 INFO:teuthology.dispatcher.supervisor:Running job 6344292 2021-08-17T16:32:18.130 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-08-17_16:14:12-rados:thrash-wip-ronenf-scrub-token-distro-basic-smithi/6344292 --name rfriedma-2021-08-17_16:14:12-rados:thrash-wip-ronenf-scrub-token-distro-basic-smithi --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} -- /home/teuthworker/archive/rfriedma-2021-08-17_16:14:12-rados:thrash-wip-ronenf-scrub-token-distro-basic-smithi/6344292/orig.config.yaml 2021-08-17T16:32:18.135 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-08-17_16:14:12-rados:thrash-wip-ronenf-scrub-token-distro-basic-smithi/6344292 2021-08-17T16:32:18.136 INFO:teuthology.dispatcher.supervisor:Job PID: 21790 2021-08-17T16:32:18.137 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-08-18T04:33:09.551 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-08-18T04:33:09.753 INFO:teuthology.kill:Killing Pids: {21790} 2021-08-18T04:33:10.990 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi205'] 2021-08-18T04:33:11.989 INFO:teuthology.kill:2021-08-18 04:33:11,989.989 INFO:teuthology.nuke:targets: 2021-08-18T04:33:11.990 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr751c+4G+7ksz1qi3Og7QYD38aXSEST1Emyi/z2QsMoIyI7HZeuDUrdqgyHbHsCecqEzmsGG68crJAad1bqG/ri+Cas8ncgrjj/WUClWRbkmTL9jJpH0Ri1TCrtXc9eY9hRGlhj2R0hUGf6qWyjNtUKk5+stg+2GT0FDRGXayVgC5GKGMZDLjBpDYF49llBf0O02ng8hjSpZUYVi63Q/hRm8gUA6mcGlQm46aJHrS5X4p9RkHiJ7JkPBwrqHfwrNhSLSbofaOkNK0w1qBOI1j1jKKGTRsFZwOwufxWIcV7U5UEPB/F5J+aa/wSxHm9acVdsuJ2x+i09UP0XpiLDFoBo0D3tdLZoXGsbqmbrwPUma4DFjHBkNTu7Csnw6N1JO24S/qqbCNd+8aX2JIzBMZ14J2UpoCvFLyhC+aOpBHT7cD1OJCk+ZKgRKub+t/Sswefk90erCtRqybdPk9XqC1ENT9XMapBlI71Bfo38GpuB6Z9lhiA1+EdUDgLuXHGs= 2021-08-18T04:33:11.990 INFO:teuthology.kill: smithi205.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm73ukpmdZ4vA9UQ3n4Wzf/T4g1b3Co/xhQ5UGWb8bTB/7qqyRa+qHK0ZIcoCowkkiOsuDncwzkB2Q7N5hQl2oIVJE/jYHo8/KiotAfPu0nFiA1QwnbQ3gRg0FPQZEQBYz0Dq+Gea1JtjswejltbPnFcoXMPWZ02iWI2xQCEm93W84Ivc9EIvvSBDZbnJsEjibvunjlXmlk++eL9BdYQsCv+Kg3LJgdXKmQrJrRXW2OCHP3K/ikWc45ANnFRXI6RiETJ1qQ/+ycWYi0/zKtClKV4w+6mZ+PTHLxsjUrLZe5SB5wd36oneU+nyL1MLeSLBNqWHIYnJSKkjf5MND9WjGTkEIHGmNMoIAFEh/yKdn3uzevxw3nBt2lAjqfegOVwWCRJXPvMzg5mJ79GMDBEGR1P+VgYeERhYdn38SoO0Nz9iJDW5kSGUQRow1SfgClT/9gHKNl4Vz5Xm2nDX90IN6KZ2BsQQzJM4hEaw4V87WS887UjWUXrCVJ3979pX0fRU= 2021-08-18T04:33:11.991 INFO:teuthology.kill:2021-08-18 04:33:11,989.989 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-18T04:33:11.992 INFO:teuthology.kill:2021-08-18 04:33:11,992.992 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-18T04:33:12.059 INFO:teuthology.kill:2021-08-18 04:33:12,059.059 INFO:teuthology.orchestra.console:Power off smithi205 2021-08-18T04:33:12.074 INFO:teuthology.kill:2021-08-18 04:33:12,074.074 INFO:teuthology.orchestra.console:Power off smithi084 2021-08-18T04:33:20.438 INFO:teuthology.kill:2021-08-18 04:33:20,436.436 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-08-18T04:33:24.540 INFO:teuthology.kill:2021-08-18 04:33:24,540.540 INFO:teuthology.orchestra.console:Power off for smithi205 completed 2021-08-18T04:33:24.876 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-08-18T04:33:24.877 INFO:teuthology.task.internal:roles: ubuntu@smithi205.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-08-18T04:33:24.877 INFO:teuthology.misc:Compressing logs... 2021-08-18T04:33:24.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-18T04:34:24.939 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-08-18T04:34:25.033 INFO:teuthology.kill:No teuthology processes running 2021-08-18T04:34:26.179 INFO:teuthology.kill:Nuking machines: ['smithi084', 'smithi205'] 2021-08-18T04:34:27.172 INFO:teuthology.kill:2021-08-18 04:34:27,171.171 INFO:teuthology.nuke:targets: 2021-08-18T04:34:27.173 INFO:teuthology.kill: smithi084.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr751c+4G+7ksz1qi3Og7QYD38aXSEST1Emyi/z2QsMoIyI7HZeuDUrdqgyHbHsCecqEzmsGG68crJAad1bqG/ri+Cas8ncgrjj/WUClWRbkmTL9jJpH0Ri1TCrtXc9eY9hRGlhj2R0hUGf6qWyjNtUKk5+stg+2GT0FDRGXayVgC5GKGMZDLjBpDYF49llBf0O02ng8hjSpZUYVi63Q/hRm8gUA6mcGlQm46aJHrS5X4p9RkHiJ7JkPBwrqHfwrNhSLSbofaOkNK0w1qBOI1j1jKKGTRsFZwOwufxWIcV7U5UEPB/F5J+aa/wSxHm9acVdsuJ2x+i09UP0XpiLDFoBo0D3tdLZoXGsbqmbrwPUma4DFjHBkNTu7Csnw6N1JO24S/qqbCNd+8aX2JIzBMZ14J2UpoCvFLyhC+aOpBHT7cD1OJCk+ZKgRKub+t/Sswefk90erCtRqybdPk9XqC1ENT9XMapBlI71Bfo38GpuB6Z9lhiA1+EdUDgLuXHGs= 2021-08-18T04:34:27.173 INFO:teuthology.kill: smithi205.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDm73ukpmdZ4vA9UQ3n4Wzf/T4g1b3Co/xhQ5UGWb8bTB/7qqyRa+qHK0ZIcoCowkkiOsuDncwzkB2Q7N5hQl2oIVJE/jYHo8/KiotAfPu0nFiA1QwnbQ3gRg0FPQZEQBYz0Dq+Gea1JtjswejltbPnFcoXMPWZ02iWI2xQCEm93W84Ivc9EIvvSBDZbnJsEjibvunjlXmlk++eL9BdYQsCv+Kg3LJgdXKmQrJrRXW2OCHP3K/ikWc45ANnFRXI6RiETJ1qQ/+ycWYi0/zKtClKV4w+6mZ+PTHLxsjUrLZe5SB5wd36oneU+nyL1MLeSLBNqWHIYnJSKkjf5MND9WjGTkEIHGmNMoIAFEh/yKdn3uzevxw3nBt2lAjqfegOVwWCRJXPvMzg5mJ79GMDBEGR1P+VgYeERhYdn38SoO0Nz9iJDW5kSGUQRow1SfgClT/9gHKNl4Vz5Xm2nDX90IN6KZ2BsQQzJM4hEaw4V87WS887UjWUXrCVJ3979pX0fRU= 2021-08-18T04:34:27.196 INFO:teuthology.kill:2021-08-18 04:34:27,196.196 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-18T04:34:27.200 INFO:teuthology.kill:2021-08-18 04:34:27,199.199 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-18T04:34:27.265 INFO:teuthology.kill:2021-08-18 04:34:27,265.265 INFO:teuthology.orchestra.console:Power off smithi205 2021-08-18T04:34:27.278 INFO:teuthology.kill:2021-08-18 04:34:27,278.278 INFO:teuthology.orchestra.console:Power off smithi084 2021-08-18T04:34:27.406 INFO:teuthology.kill:2021-08-18 04:34:27,405.405 INFO:teuthology.orchestra.console:Power off for smithi205 completed 2021-08-18T04:34:27.419 INFO:teuthology.kill:2021-08-18 04:34:27,419.419 INFO:teuthology.orchestra.console:Power off for smithi084 completed 2021-08-18T04:34:27.549 INFO:teuthology.kill:2021-08-18 04:34:27,549.549 INFO:teuthology.lock.ops:unlocked smithi205.front.sepia.ceph.com 2021-08-18T04:34:27.558 INFO:teuthology.kill:2021-08-18 04:34:27,558.558 INFO:teuthology.lock.ops:unlocked smithi084.front.sepia.ceph.com 2021-08-18T04:36:27.840 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-08-18T04:36:27.930 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-08-18T04:36:27.931 INFO:teuthology.nuke:Checking targets against current locks 2021-08-18T04:36:28.058 INFO:teuthology.nuke:Not nuking smithi084.front.sepia.ceph.com because description doesn't match 2021-08-18T04:36:28.274 INFO:teuthology.nuke:Not nuking smithi205.front.sepia.ceph.com because description doesn't match