2021-06-29T08:41:58.631 INFO:root:teuthology version: 1.1.0-26631ede 2021-06-29T08:41:58.914 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging 2021-06-29T08:41:58.915 INFO:teuthology.lock.ops:Updating [ubuntu@smithi078.front.sepia.ceph.com]: reset os type and version on server 2021-06-29T08:41:58.915 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2021-06-29T08:41:58.932 INFO:teuthology.lock.ops:Node 'ubuntu@smithi078.front.sepia.ceph.com' reimaging is complete 2021-06-29T08:41:58.932 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi200.front.sepia.ceph.com' reimaging 2021-06-29T08:41:58.932 INFO:teuthology.lock.ops:Updating [ubuntu@smithi200.front.sepia.ceph.com]: reset os type and version on server 2021-06-29T08:41:58.933 INFO:teuthology.lock.ops:Updating smithi200.front.sepia.ceph.com on lock server 2021-06-29T08:41:58.952 INFO:teuthology.lock.ops:Node 'ubuntu@smithi200.front.sepia.ceph.com' reimaging is complete 2021-06-29T08:41:59.147 INFO:teuthology.provision.fog.smithi078:Scheduling deploy of centos 8.3 2021-06-29T08:41:59.185 INFO:teuthology.provision.fog.smithi200:Scheduling deploy of centos 8.3 2021-06-29T08:41:59.624 INFO:teuthology.orchestra.console:Power off smithi078 2021-06-29T08:41:59.838 INFO:teuthology.orchestra.console:Power off smithi200 2021-06-29T08:42:08.334 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-06-29T08:42:08.439 INFO:teuthology.orchestra.console:Power on smithi200 2021-06-29T08:42:12.377 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-06-29T08:42:12.485 INFO:teuthology.orchestra.console:Power on smithi078 2021-06-29T08:42:16.903 INFO:teuthology.orchestra.console:Power on for smithi200 completed 2021-06-29T08:42:17.020 INFO:teuthology.provision.fog.smithi200:Waiting for deploy to finish 2021-06-29T08:42:20.941 INFO:teuthology.orchestra.console:Power on for smithi078 completed 2021-06-29T08:42:21.046 INFO:teuthology.provision.fog.smithi078:Waiting for deploy to finish 2021-06-29T08:46:24.569 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-29T08:46:44.693 INFO:teuthology.provision.fog.smithi078:Node is ready 2021-06-29T08:46:44.712 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078.front.sepia.ceph.com 2021-06-29T08:46:44.769 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78 smithi078.front.sepia.ceph.com smithi078 2021-06-29T08:46:45.122 INFO:teuthology.provision.fog.smithi078:Deploy complete! 2021-06-29T08:46:50.414 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-06-29T08:46:50.705 INFO:teuthology.provision.fog.smithi200:Node is ready 2021-06-29T08:46:50.719 INFO:teuthology.orchestra.run.smithi200.stdout:smithi200.front.sepia.ceph.com 2021-06-29T08:46:50.774 INFO:teuthology.orchestra.run.smithi200.stdout:172.21.15.200 smithi200.front.sepia.ceph.com smithi200 2021-06-29T08:46:51.089 INFO:teuthology.provision.fog.smithi200:Deploy complete! 2021-06-29T08:46:51.208 INFO:teuthology.lock.ops:Checking smithi078.front.sepia.ceph.com 2021-06-29T08:46:51.209 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-29T08:46:51.225 INFO:teuthology.lock.ops:Checking smithi200.front.sepia.ceph.com 2021-06-29T08:46:51.226 INFO:teuthology.lock.ops:New key found. Updating... 2021-06-29T08:46:51.240 INFO:teuthology.lock.ops:Updating [smithi078.front.sepia.ceph.com]: set os type and version on server 2021-06-29T08:46:51.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-29T08:46:51.531 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2021-06-29T08:46:51.585 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Linux" 2021-06-29T08:46:51.585 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8" 2021-06-29T08:46:51.585 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2021-06-29T08:46:51.586 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2021-06-29T08:46:51.586 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8" 2021-06-29T08:46:51.586 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2021-06-29T08:46:51.587 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-29T08:46:51.587 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2021-06-29T08:46:51.587 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-29T08:46:51.587 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2021-06-29T08:46:51.588 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-29T08:46:51.588 INFO:teuthology.orchestra.run.smithi078.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-29T08:46:51.588 INFO:teuthology.orchestra.run.smithi078.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-29T08:46:51.589 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2021-06-29T08:46:51.604 INFO:teuthology.lock.ops:Updating [smithi200.front.sepia.ceph.com]: set os type and version on server 2021-06-29T08:46:51.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-29T08:46:51.882 INFO:teuthology.orchestra.run.smithi200.stdout:x86_64 2021-06-29T08:46:51.936 INFO:teuthology.orchestra.run.smithi200.stdout:NAME="CentOS Linux" 2021-06-29T08:46:51.936 INFO:teuthology.orchestra.run.smithi200.stdout:VERSION="8" 2021-06-29T08:46:51.936 INFO:teuthology.orchestra.run.smithi200.stdout:ID="centos" 2021-06-29T08:46:51.937 INFO:teuthology.orchestra.run.smithi200.stdout:ID_LIKE="rhel fedora" 2021-06-29T08:46:51.937 INFO:teuthology.orchestra.run.smithi200.stdout:VERSION_ID="8" 2021-06-29T08:46:51.937 INFO:teuthology.orchestra.run.smithi200.stdout:PLATFORM_ID="platform:el8" 2021-06-29T08:46:51.937 INFO:teuthology.orchestra.run.smithi200.stdout:PRETTY_NAME="CentOS Linux 8" 2021-06-29T08:46:51.938 INFO:teuthology.orchestra.run.smithi200.stdout:ANSI_COLOR="0;31" 2021-06-29T08:46:51.938 INFO:teuthology.orchestra.run.smithi200.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-06-29T08:46:51.938 INFO:teuthology.orchestra.run.smithi200.stdout:HOME_URL="https://centos.org/" 2021-06-29T08:46:51.938 INFO:teuthology.orchestra.run.smithi200.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-06-29T08:46:51.939 INFO:teuthology.orchestra.run.smithi200.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-06-29T08:46:51.939 INFO:teuthology.orchestra.run.smithi200.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-06-29T08:46:51.939 INFO:teuthology.lock.ops:Updating smithi200.front.sepia.ceph.com on lock server 2021-06-29T08:46:52.033 INFO:teuthology.dispatcher.supervisor:Running job 6205785 2021-06-29T08:46:52.034 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6205785 --name rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big} -- /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6205785/orig.config.yaml 2021-06-29T08:46:52.041 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distro-basic-smithi/6205785 2021-06-29T08:46:52.042 INFO:teuthology.dispatcher.supervisor:Job PID: 27274 2021-06-29T08:46:52.042 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-06-29T20:48:03.967 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-06-29T20:48:05.509 INFO:teuthology.kill:Killing Pids: {27274} 2021-06-29T20:48:07.225 INFO:teuthology.kill:Nuking machines: ['smithi078', 'smithi200'] 2021-06-29T20:48:08.242 INFO:teuthology.kill:2021-06-29 20:48:08,242.242 INFO:teuthology.nuke:targets: 2021-06-29T20:48:08.243 INFO:teuthology.kill: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHUXlvgjeyUxCwUB6Z4BT7JQ60kIkceRtFejL1DZdNvSPnC5seUU1qZj+ozvP9eOHgKYoz4Y014Z2gMd7PJnbKkTmD1JTz/NOZVONwGpcUlMdcOul4rKtiTSFh4HAw2uOTC7puYx0Wc5JqXztcYhjAYWpwPRCarpqnZJrSRN7sNXHqEemW3Qm0eW2sc4d2EO0vson9TmpbNHyIVXONGIGgExe672bL5HvbVpbJYr1eTw5YXuZ/M67rHtrOzNauG+nWWdIJ4uI0tGCxWwJxXaDvV2JtJwCDGOaHFjKkH5NjgwBLZV+V91ButnGiQTqfm/QcCPCey6pgbjgetCva0TzCnW5rpmai+LbRZgFfuFtREkzKzRUTEcW30qZXA6ZIOhPpZRZWqUoLGlnbTnJ6x3GkEePkdIQvH+QOv6F53hvkLjErsr0ZJBOhjLT1UEKsRRw94xaw41gWHKyblB4qirqVXmAakpR1ZjJ9Ij7sdg/YhQ9pxVIWWF8fUfevqOboM8c= 2021-06-29T20:48:08.246 INFO:teuthology.kill: smithi200.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLONsV8Sy9FZs5fZ3NyxZE220+Wxr4RnOR92m3yPSt3+coB/koNInECljmEAHcEsw4TbZaADGzNJDRDopt/LUqtsnm7Zjr467+BA0Zw5742qMy/Hoz6CQufUhMmdNAfw8AXm7x13vDgD0/uINAbgpoUMnzIGBpONURgo9NzagXPw1Nh2sUSLmAOcWuJsGBgEA045QvZh9zM9X83HXquYEcg5xOJzslr0nC8Soe2NyLZJarms1MNigea6ikopSHS8jdrszPBo7twiNOdjoddXVMmq5P04APb3yEo378L4MlOovTjI8p6pokzH9pRoddy7SRyLc1hgBosb7MbUK7u9uQYPmbHPwnZCj1YGlVqwDe8nw3DsXl/uizykm/YNLsz/Wj+V8nRHyj4Mm7MnSR3j/X3xY5caw2ebUlTGH5z7i6kSjwd8LBGB6nIIK9lwMy9zA/hBOTYFR25/NeHgd+lXUq5T+1GXyjXkp7Y08dL/zsRGvowMmYSNRZIe2yNTaxTeM= 2021-06-29T20:48:08.246 INFO:teuthology.kill:2021-06-29 20:48:08,244.244 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T20:48:08.246 INFO:teuthology.kill:2021-06-29 20:48:08,246.246 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T20:48:08.312 INFO:teuthology.kill:2021-06-29 20:48:08,312.312 INFO:teuthology.orchestra.console:Power off smithi078 2021-06-29T20:48:08.330 INFO:teuthology.kill:2021-06-29 20:48:08,328.328 INFO:teuthology.orchestra.console:Power off smithi200 2021-06-29T20:48:16.710 INFO:teuthology.kill:2021-06-29 20:48:16,709.709 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-06-29T20:48:16.717 INFO:teuthology.kill:2021-06-29 20:48:16,717.717 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-06-29T20:48:17.236 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-06-29T20:48:17.255 INFO:teuthology.task.internal:roles: ubuntu@smithi200.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-06-29T20:48:17.276 INFO:teuthology.misc:Compressing logs... 2021-06-29T20:48:17.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-06-29T20:49:17.480 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_26631ede9027a5643f2e47dacc5af6f1834aac45/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-06-29T20:49:17.613 INFO:teuthology.kill:No teuthology processes running 2021-06-29T20:49:18.752 INFO:teuthology.kill:Nuking machines: ['smithi078', 'smithi200'] 2021-06-29T20:49:19.698 INFO:teuthology.kill:2021-06-29 20:49:19,698.698 INFO:teuthology.nuke:targets: 2021-06-29T20:49:19.699 INFO:teuthology.kill: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHUXlvgjeyUxCwUB6Z4BT7JQ60kIkceRtFejL1DZdNvSPnC5seUU1qZj+ozvP9eOHgKYoz4Y014Z2gMd7PJnbKkTmD1JTz/NOZVONwGpcUlMdcOul4rKtiTSFh4HAw2uOTC7puYx0Wc5JqXztcYhjAYWpwPRCarpqnZJrSRN7sNXHqEemW3Qm0eW2sc4d2EO0vson9TmpbNHyIVXONGIGgExe672bL5HvbVpbJYr1eTw5YXuZ/M67rHtrOzNauG+nWWdIJ4uI0tGCxWwJxXaDvV2JtJwCDGOaHFjKkH5NjgwBLZV+V91ButnGiQTqfm/QcCPCey6pgbjgetCva0TzCnW5rpmai+LbRZgFfuFtREkzKzRUTEcW30qZXA6ZIOhPpZRZWqUoLGlnbTnJ6x3GkEePkdIQvH+QOv6F53hvkLjErsr0ZJBOhjLT1UEKsRRw94xaw41gWHKyblB4qirqVXmAakpR1ZjJ9Ij7sdg/YhQ9pxVIWWF8fUfevqOboM8c= 2021-06-29T20:49:19.700 INFO:teuthology.kill: smithi200.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDLONsV8Sy9FZs5fZ3NyxZE220+Wxr4RnOR92m3yPSt3+coB/koNInECljmEAHcEsw4TbZaADGzNJDRDopt/LUqtsnm7Zjr467+BA0Zw5742qMy/Hoz6CQufUhMmdNAfw8AXm7x13vDgD0/uINAbgpoUMnzIGBpONURgo9NzagXPw1Nh2sUSLmAOcWuJsGBgEA045QvZh9zM9X83HXquYEcg5xOJzslr0nC8Soe2NyLZJarms1MNigea6ikopSHS8jdrszPBo7twiNOdjoddXVMmq5P04APb3yEo378L4MlOovTjI8p6pokzH9pRoddy7SRyLc1hgBosb7MbUK7u9uQYPmbHPwnZCj1YGlVqwDe8nw3DsXl/uizykm/YNLsz/Wj+V8nRHyj4Mm7MnSR3j/X3xY5caw2ebUlTGH5z7i6kSjwd8LBGB6nIIK9lwMy9zA/hBOTYFR25/NeHgd+lXUq5T+1GXyjXkp7Y08dL/zsRGvowMmYSNRZIe2yNTaxTeM= 2021-06-29T20:49:19.720 INFO:teuthology.kill:2021-06-29 20:49:19,720.720 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T20:49:19.723 INFO:teuthology.kill:2021-06-29 20:49:19,723.723 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-06-29T20:49:19.781 INFO:teuthology.kill:2021-06-29 20:49:19,781.781 INFO:teuthology.orchestra.console:Power off smithi200 2021-06-29T20:49:19.797 INFO:teuthology.kill:2021-06-29 20:49:19,796.796 INFO:teuthology.orchestra.console:Power off smithi078 2021-06-29T20:49:19.928 INFO:teuthology.kill:2021-06-29 20:49:19,927.927 INFO:teuthology.orchestra.console:Power off for smithi200 completed 2021-06-29T20:49:19.939 INFO:teuthology.kill:2021-06-29 20:49:19,939.939 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2021-06-29T20:49:20.082 INFO:teuthology.kill:2021-06-29 20:49:20,082.082 INFO:teuthology.lock.ops:unlocked smithi200.front.sepia.ceph.com 2021-06-29T20:49:20.084 INFO:teuthology.kill:2021-06-29 20:49:20,084.084 INFO:teuthology.lock.ops:unlocked smithi078.front.sepia.ceph.com 2021-06-29T20:51:21.483 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-06-29T20:51:21.620 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-06-29T20:51:21.628 INFO:teuthology.nuke:Checking targets against current locks 2021-06-29T20:51:21.850 INFO:teuthology.nuke:Not nuking smithi200.front.sepia.ceph.com because description doesn't match 2021-06-29T20:51:21.850 INFO:teuthology.nuke:Not nuking smithi078.front.sepia.ceph.com because description doesn't match