2021-06-23T20:17:27.505 INFO:root:teuthology version: 1.1.0-26631ede 2021-06-23T20:17:28.625 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging 2021-06-23T20:17:28.626 INFO:teuthology.lock.ops:Updating [ubuntu@smithi008.front.sepia.ceph.com]: reset os type and version on server 2021-06-23T20:17:28.626 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2021-06-23T20:17:28.640 INFO:teuthology.lock.ops:Node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging is complete 2021-06-23T20:17:28.640 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi148.front.sepia.ceph.com' reimaging 2021-06-23T20:17:28.640 INFO:teuthology.lock.ops:Updating [ubuntu@smithi148.front.sepia.ceph.com]: reset os type and version on server 2021-06-23T20:17:28.641 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2021-06-23T20:17:28.656 INFO:teuthology.lock.ops:Node 'ubuntu@smithi148.front.sepia.ceph.com' reimaging is complete 2021-06-23T20:17:28.866 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of centos 8.3 2021-06-23T20:17:28.871 INFO:teuthology.provision.fog.smithi148:Scheduling deploy of centos 8.3 2021-06-23T20:17:29.218 INFO:teuthology.orchestra.console:Power off smithi148 2021-06-23T20:17:29.254 INFO:teuthology.orchestra.console:Power off smithi008 2021-06-23T20:17:37.589 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2021-06-23T20:17:37.617 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-06-23T20:17:37.691 INFO:teuthology.orchestra.console:Power on smithi148 2021-06-23T20:17:37.719 INFO:teuthology.orchestra.console:Power on smithi008 2021-06-23T20:17:46.057 INFO:teuthology.orchestra.console:Power on for smithi148 completed 2021-06-23T20:17:46.079 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2021-06-23T20:17:46.159 INFO:teuthology.provision.fog.smithi148:Waiting for deploy to finish 2021-06-23T20:17:46.180 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2021-06-23T20:22:18.090 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-23T20:22:18.382 INFO:teuthology.provision.fog.smithi008:Node is ready 2021-06-23T20:22:18.398 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008.front.sepia.ceph.com 2021-06-23T20:22:18.453 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.8 smithi008.front.sepia.ceph.com smithi008 2021-06-23T20:22:18.785 INFO:teuthology.provision.fog.smithi008:Deploy complete! 2021-06-23T20:22:33.112 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-23T20:22:33.460 INFO:teuthology.provision.fog.smithi148:Node is ready 2021-06-23T20:22:33.475 INFO:teuthology.orchestra.run.smithi148.stdout:smithi148.front.sepia.ceph.com 2021-06-23T20:22:33.531 INFO:teuthology.orchestra.run.smithi148.stdout:172.21.15.148 smithi148.front.sepia.ceph.com smithi148 2021-06-23T20:22:33.841 INFO:teuthology.provision.fog.smithi148:Deploy complete! 2021-06-23T20:22:33.965 INFO:teuthology.lock.ops:Checking smithi008.front.sepia.ceph.com 2021-06-23T20:22:33.966 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-23T20:22:33.986 INFO:teuthology.lock.ops:Checking smithi148.front.sepia.ceph.com 2021-06-23T20:22:33.986 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-23T20:22:34.004 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: set os type and version on server 2021-06-23T20:22:34.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-23T20:22:34.300 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2021-06-23T20:22:34.355 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="CentOS Linux" 2021-06-23T20:22:34.356 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="8" 2021-06-23T20:22:34.356 INFO:teuthology.orchestra.run.smithi008.stdout:ID="centos" 2021-06-23T20:22:34.356 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="rhel fedora" 2021-06-23T20:22:34.357 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="8" 2021-06-23T20:22:34.357 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el8" 2021-06-23T20:22:34.357 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-23T20:22:34.357 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2021-06-23T20:22:34.357 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-23T20:22:34.358 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://centos.org/" 2021-06-23T20:22:34.358 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-23T20:22:34.358 INFO:teuthology.orchestra.run.smithi008.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-23T20:22:34.358 INFO:teuthology.orchestra.run.smithi008.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-23T20:22:34.360 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2021-06-23T20:22:34.376 INFO:teuthology.lock.ops:Updating [smithi148.front.sepia.ceph.com]: set os type and version on server 2021-06-23T20:22:34.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-23T20:22:34.660 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2021-06-23T20:22:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Linux" 2021-06-23T20:22:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="8" 2021-06-23T20:22:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2021-06-23T20:22:34.715 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2021-06-23T20:22:34.716 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="8" 2021-06-23T20:22:34.716 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el8" 2021-06-23T20:22:34.716 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-23T20:22:34.716 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2021-06-23T20:22:34.716 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-23T20:22:34.717 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2021-06-23T20:22:34.717 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-23T20:22:34.717 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-23T20:22:34.717 INFO:teuthology.orchestra.run.smithi148.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-23T20:22:34.718 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2021-06-23T20:22:34.801 INFO:teuthology.dispatcher.supervisor:Running job 6187460 2021-06-23T20:22:34.802 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2021-06-23_19:31:08-rados:thrash-master-distro-basic-smithi/6187460 --name nojha-2021-06-23_19:31:08-rados:thrash-master-distro-basic-smithi --description rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced} -- /home/teuthworker/archive/nojha-2021-06-23_19:31:08-rados:thrash-master-distro-basic-smithi/6187460/orig.config.yaml 2021-06-23T20:22:34.806 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2021-06-23_19:31:08-rados:thrash-master-distro-basic-smithi/6187460 2021-06-23T20:22:34.807 INFO:teuthology.dispatcher.supervisor:Job PID: 14440 2021-06-23T20:22:34.807 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-24T08:23:27.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-24T08:23:27.856 INFO:teuthology.kill:Killing Pids: {14440} 2021-06-24T08:23:29.087 INFO:teuthology.kill:Nuking machines: ['smithi008', 'smithi148'] 2021-06-24T08:23:30.092 INFO:teuthology.kill:2021-06-24 08:23:30,092.092 INFO:teuthology.nuke:targets: 2021-06-24T08:23:30.093 INFO:teuthology.kill: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqLm5/S3OhkGgchb37sGBQVbkv2jw1BRVr72LEg9EDav94CIypCLac5KGkZFd2jMtkNE/EwfEN50E9MbmMQ/wsOnDudlMjoSWTXgk8F1TzaPvsygWvMrk/h+00sG+Dzjhke5jb5+zGW/0hQqodkolPMtXAttz1yjjOsf3KOp/MtU4TYYPR+5bikXvAG9F4+OueYAaBZN5Wz4YemUD1Ah4coDItOI/22CguGawdChm6UPmppZRIxBblkO25lRaYzC3cQMSkbO91AMyEEZSE8MLW3XiScDVgqkylZFmG1tSsxQbxqUAN+HvjWpLCCFCz+24KznBbTUCTZVAXwnxuzJmI+5iEaq7pdo7r08QwmmyGoGWyn0Mt7I352YPIDfZSsBA+ZoSbgNSfXkbIjocXDvleHZsGfjmfTPC8uvJmrBgLZ/i7DfRK8EYahzqaNcHoaS8vOZOV+fV610mwW4uSI1684k4W+y6a7bIizb4FICto2maPQDXdFzPvarUwEwDl0Lc= 2021-06-24T08:23:30.093 INFO:teuthology.kill: smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiu3JZ4xp0lUIuX2YzCOhCdhs9frX0C3kp+0v8l8hwwaCR7mVh0jCmBDumgwze+WAQ/rYVtGBIco+XZvvlJTzzIeYW8dovP6zdwH5w3Xzn724gOH6T4Hk3dUKtKtxi9OyT4R/7XBfKQ+igFK0/zIMDfY1mpugWtpmcTfsIdr/6kY1oc7aoYKbIIQZ3raY2J+l5EckxIY2tunwjVTNVlx1oaI6FoHpr4U08botuk+tp2twEhaxlRdY2BSGBviII22lVasWEOln/kG3knEAdT6tJ7ZvM/uEqzTyQdUJ5bPxohG6s87JTjL+GM7CUN7UJMUM/jq53SYEeRoqQtdpERyIr0Dv9Dvaj99np3+eR/sovxgLjVy2zwxck4exBHBL0LB52W4hslGm2RfF209w4IvpmeJoqRF+3QPwkonoxaGD+Cc/nVCSL90h8awjt7awiAzEBXmu7y8E8i0dWvsFuqdzpKVif0t6WV7VzAr3EGFmyqF6pg+tLhoK7/bR8f9fA5ak= 2021-06-24T08:23:30.094 INFO:teuthology.kill:2021-06-24 08:23:30,092.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-24T08:23:30.095 INFO:teuthology.kill:2021-06-24 08:23:30,095.095 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-24T08:23:30.159 INFO:teuthology.kill:2021-06-24 08:23:30,158.158 INFO:teuthology.orchestra.console:Power off smithi008 2021-06-24T08:23:30.176 INFO:teuthology.kill:2021-06-24 08:23:30,175.175 INFO:teuthology.orchestra.console:Power off smithi148 2021-06-24T08:23:38.534 INFO:teuthology.kill:2021-06-24 08:23:38,533.533 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-06-24T08:23:42.655 INFO:teuthology.kill:2021-06-24 08:23:42,655.655 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2021-06-24T08:23:42.880 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-06-24T08:23:42.881 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-06-24T08:23:42.881 INFO:teuthology.misc:Compressing logs... 2021-06-24T08:23:42.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-24T08:24:17.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-24T08:24:17.422 INFO:teuthology.kill:No teuthology processes running 2021-06-24T08:24:18.593 INFO:teuthology.kill:Nuking machines: ['smithi008', 'smithi148'] 2021-06-24T08:24:19.683 INFO:teuthology.kill:2021-06-24 08:24:19,683.683 INFO:teuthology.nuke:targets: 2021-06-24T08:24:19.684 INFO:teuthology.kill: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqLm5/S3OhkGgchb37sGBQVbkv2jw1BRVr72LEg9EDav94CIypCLac5KGkZFd2jMtkNE/EwfEN50E9MbmMQ/wsOnDudlMjoSWTXgk8F1TzaPvsygWvMrk/h+00sG+Dzjhke5jb5+zGW/0hQqodkolPMtXAttz1yjjOsf3KOp/MtU4TYYPR+5bikXvAG9F4+OueYAaBZN5Wz4YemUD1Ah4coDItOI/22CguGawdChm6UPmppZRIxBblkO25lRaYzC3cQMSkbO91AMyEEZSE8MLW3XiScDVgqkylZFmG1tSsxQbxqUAN+HvjWpLCCFCz+24KznBbTUCTZVAXwnxuzJmI+5iEaq7pdo7r08QwmmyGoGWyn0Mt7I352YPIDfZSsBA+ZoSbgNSfXkbIjocXDvleHZsGfjmfTPC8uvJmrBgLZ/i7DfRK8EYahzqaNcHoaS8vOZOV+fV610mwW4uSI1684k4W+y6a7bIizb4FICto2maPQDXdFzPvarUwEwDl0Lc= 2021-06-24T08:24:19.685 INFO:teuthology.kill: smithi148.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiu3JZ4xp0lUIuX2YzCOhCdhs9frX0C3kp+0v8l8hwwaCR7mVh0jCmBDumgwze+WAQ/rYVtGBIco+XZvvlJTzzIeYW8dovP6zdwH5w3Xzn724gOH6T4Hk3dUKtKtxi9OyT4R/7XBfKQ+igFK0/zIMDfY1mpugWtpmcTfsIdr/6kY1oc7aoYKbIIQZ3raY2J+l5EckxIY2tunwjVTNVlx1oaI6FoHpr4U08botuk+tp2twEhaxlRdY2BSGBviII22lVasWEOln/kG3knEAdT6tJ7ZvM/uEqzTyQdUJ5bPxohG6s87JTjL+GM7CUN7UJMUM/jq53SYEeRoqQtdpERyIr0Dv9Dvaj99np3+eR/sovxgLjVy2zwxck4exBHBL0LB52W4hslGm2RfF209w4IvpmeJoqRF+3QPwkonoxaGD+Cc/nVCSL90h8awjt7awiAzEBXmu7y8E8i0dWvsFuqdzpKVif0t6WV7VzAr3EGFmyqF6pg+tLhoK7/bR8f9fA5ak= 2021-06-24T08:24:19.706 INFO:teuthology.kill:2021-06-24 08:24:19,706.706 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-24T08:24:19.709 INFO:teuthology.kill:2021-06-24 08:24:19,709.709 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-24T08:24:19.770 INFO:teuthology.kill:2021-06-24 08:24:19,769.769 INFO:teuthology.orchestra.console:Power off smithi008 2021-06-24T08:24:19.787 INFO:teuthology.kill:2021-06-24 08:24:19,787.787 INFO:teuthology.orchestra.console:Power off smithi148 2021-06-24T08:24:19.920 INFO:teuthology.kill:2021-06-24 08:24:19,919.919 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2021-06-24T08:24:19.932 INFO:teuthology.kill:2021-06-24 08:24:19,932.932 INFO:teuthology.orchestra.console:Power off for smithi148 completed 2021-06-24T08:24:20.064 INFO:teuthology.kill:2021-06-24 08:24:20,063.063 INFO:teuthology.lock.ops:unlocked smithi008.front.sepia.ceph.com 2021-06-24T08:24:20.076 INFO:teuthology.kill:2021-06-24 08:24:20,076.076 INFO:teuthology.lock.ops:unlocked smithi148.front.sepia.ceph.com 2021-06-24T08:26:20.381 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-24T08:26:20.476 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-24T08:26:20.477 INFO:teuthology.nuke:Checking targets against current locks 2021-06-24T08:26:20.544 INFO:teuthology.nuke:Not nuking smithi008.front.sepia.ceph.com because description doesn't match 2021-06-24T08:26:20.544 INFO:teuthology.nuke:Not nuking smithi148.front.sepia.ceph.com because description doesn't match