2021-05-27T01:30:55.111 INFO:root:teuthology version: 1.1.0-d92ec140 2021-05-27T01:30:55.227 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi028.front.sepia.ceph.com' reimaging 2021-05-27T01:30:55.228 INFO:teuthology.lock.ops:Updating [ubuntu@smithi028.front.sepia.ceph.com]: reset os type and version on server 2021-05-27T01:30:55.228 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2021-05-27T01:30:55.244 INFO:teuthology.lock.ops:Node 'ubuntu@smithi028.front.sepia.ceph.com' reimaging is complete 2021-05-27T01:30:55.245 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi058.front.sepia.ceph.com' reimaging 2021-05-27T01:30:55.246 INFO:teuthology.lock.ops:Updating [ubuntu@smithi058.front.sepia.ceph.com]: reset os type and version on server 2021-05-27T01:30:55.246 INFO:teuthology.lock.ops:Updating smithi058.front.sepia.ceph.com on lock server 2021-05-27T01:30:55.260 INFO:teuthology.lock.ops:Node 'ubuntu@smithi058.front.sepia.ceph.com' reimaging is complete 2021-05-27T01:30:55.464 INFO:teuthology.provision.fog.smithi028:Scheduling deploy of centos 8.2 2021-05-27T01:30:55.467 INFO:teuthology.provision.fog.smithi058:Scheduling deploy of centos 8.2 2021-05-27T01:30:55.780 INFO:teuthology.orchestra.console:Power off smithi058 2021-05-27T01:30:55.815 INFO:teuthology.orchestra.console:Power off smithi028 2021-05-27T01:31:04.149 INFO:teuthology.orchestra.console:Power off for smithi058 completed 2021-05-27T01:31:04.251 INFO:teuthology.orchestra.console:Power on smithi058 2021-05-27T01:31:08.292 INFO:teuthology.orchestra.console:Power off for smithi028 completed 2021-05-27T01:31:08.394 INFO:teuthology.orchestra.console:Power on smithi028 2021-05-27T01:31:12.618 INFO:teuthology.orchestra.console:Power on for smithi058 completed 2021-05-27T01:31:12.719 INFO:teuthology.provision.fog.smithi058:Waiting for deploy to finish 2021-05-27T01:31:16.760 INFO:teuthology.orchestra.console:Power on for smithi028 completed 2021-05-27T01:31:16.861 INFO:teuthology.provision.fog.smithi028:Waiting for deploy to finish 2021-05-27T01:35:20.537 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-27T01:35:20.863 INFO:teuthology.provision.fog.smithi058:Node is ready 2021-05-27T01:35:20.885 INFO:teuthology.orchestra.run.smithi058.stdout:smithi058.front.sepia.ceph.com 2021-05-27T01:35:20.949 INFO:teuthology.orchestra.run.smithi058.stdout:172.21.15.58 smithi058.front.sepia.ceph.com smithi058 2021-05-27T01:35:21.287 INFO:teuthology.provision.fog.smithi058:Deploy complete! 2021-05-27T01:35:24.465 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-05-27T01:35:24.779 INFO:teuthology.provision.fog.smithi028:Node is ready 2021-05-27T01:35:24.795 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028.front.sepia.ceph.com 2021-05-27T01:35:24.850 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28 smithi028.front.sepia.ceph.com smithi028 2021-05-27T01:35:25.175 INFO:teuthology.provision.fog.smithi028:Deploy complete! 2021-05-27T01:35:25.296 INFO:teuthology.lock.ops:Checking smithi028.front.sepia.ceph.com 2021-05-27T01:35:25.297 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-27T01:35:25.312 INFO:teuthology.lock.ops:Checking smithi058.front.sepia.ceph.com 2021-05-27T01:35:25.312 INFO:teuthology.lock.ops:New key found. Updating... 2021-05-27T01:35:25.327 INFO:teuthology.lock.ops:Updating [smithi028.front.sepia.ceph.com]: set os type and version on server 2021-05-27T01:35:25.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T01:35:25.598 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2021-05-27T01:35:25.652 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Linux" 2021-05-27T01:35:25.653 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="8 (Core)" 2021-05-27T01:35:25.653 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2021-05-27T01:35:25.653 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2021-05-27T01:35:25.653 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="8" 2021-05-27T01:35:25.654 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el8" 2021-05-27T01:35:25.654 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-27T01:35:25.654 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2021-05-27T01:35:25.654 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-27T01:35:25.655 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://www.centos.org/" 2021-05-27T01:35:25.655 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-27T01:35:25.656 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-27T01:35:25.656 INFO:teuthology.orchestra.run.smithi028.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-27T01:35:25.656 INFO:teuthology.orchestra.run.smithi028.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-27T01:35:25.656 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-27T01:35:25.656 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-27T01:35:25.657 INFO:teuthology.orchestra.run.smithi028.stdout: 2021-05-27T01:35:25.658 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2021-05-27T01:35:25.676 INFO:teuthology.lock.ops:Updating [smithi058.front.sepia.ceph.com]: set os type and version on server 2021-05-27T01:35:25.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T01:35:26.040 INFO:teuthology.orchestra.run.smithi058.stdout:x86_64 2021-05-27T01:35:26.099 INFO:teuthology.orchestra.run.smithi058.stdout:NAME="CentOS Linux" 2021-05-27T01:35:26.099 INFO:teuthology.orchestra.run.smithi058.stdout:VERSION="8 (Core)" 2021-05-27T01:35:26.099 INFO:teuthology.orchestra.run.smithi058.stdout:ID="centos" 2021-05-27T01:35:26.099 INFO:teuthology.orchestra.run.smithi058.stdout:ID_LIKE="rhel fedora" 2021-05-27T01:35:26.100 INFO:teuthology.orchestra.run.smithi058.stdout:VERSION_ID="8" 2021-05-27T01:35:26.100 INFO:teuthology.orchestra.run.smithi058.stdout:PLATFORM_ID="platform:el8" 2021-05-27T01:35:26.100 INFO:teuthology.orchestra.run.smithi058.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-05-27T01:35:26.100 INFO:teuthology.orchestra.run.smithi058.stdout:ANSI_COLOR="0;31" 2021-05-27T01:35:26.100 INFO:teuthology.orchestra.run.smithi058.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-05-27T01:35:26.101 INFO:teuthology.orchestra.run.smithi058.stdout:HOME_URL="https://www.centos.org/" 2021-05-27T01:35:26.101 INFO:teuthology.orchestra.run.smithi058.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-05-27T01:35:26.101 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-05-27T01:35:26.102 INFO:teuthology.orchestra.run.smithi058.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-05-27T01:35:26.102 INFO:teuthology.orchestra.run.smithi058.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-05-27T01:35:26.102 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-05-27T01:35:26.102 INFO:teuthology.orchestra.run.smithi058.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-05-27T01:35:26.103 INFO:teuthology.orchestra.run.smithi058.stdout: 2021-05-27T01:35:26.103 INFO:teuthology.lock.ops:Updating smithi058.front.sepia.ceph.com on lock server 2021-05-27T01:35:26.208 INFO:teuthology.dispatcher.supervisor:Running job 6133192 2021-05-27T01:35:26.208 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2021-05-25_04:17:02-fs-pacific-distro-basic-smithi/6133192 --name teuthology-2021-05-25_04:17:02-fs-pacific-distro-basic-smithi --description fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} -- /home/teuthworker/archive/teuthology-2021-05-25_04:17:02-fs-pacific-distro-basic-smithi/6133192/orig.config.yaml 2021-05-27T01:35:26.211 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2021-05-25_04:17:02-fs-pacific-distro-basic-smithi/6133192 2021-05-27T01:35:26.212 INFO:teuthology.dispatcher.supervisor:Job PID: 32204 2021-05-27T01:35:26.213 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-05-27T13:36:12.563 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-05-27T13:36:12.767 INFO:teuthology.kill:Killing Pids: {32204} 2021-05-27T13:36:13.762 INFO:teuthology.kill:Nuking machines: ['smithi028', 'smithi058'] 2021-05-27T13:36:14.587 INFO:teuthology.kill:2021-05-27 13:36:14,586.586 INFO:teuthology.nuke:targets: 2021-05-27T13:36:14.587 INFO:teuthology.kill: smithi028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC38bO+jXR9ypoNHg7VM+sC4QALcqhWv8aMdzOPPAaB6M0hO+F5Ib7XnW/dIo+80bVyEQTliEjpsrjAa4WRFVSoBIeXXCgmr/SK7IMRoumYRE5Hv9qvXC47vULMfgBDlEdskw1UVJq/F2Zjj/PhOMVOE5AohfHSv5/lvO8izYzakYTMlXwoRI3jElDfI3bF+Y17k/YlF31/iGniebdX7rrZVd4QwWQrxr85JEYwmWTFDxyyuV9UXfdjcU/KBEB0FA5mPTYaTWGbNGu7ycpdjOnEwehs9whRigNXrLw+t6LfKrf56REh9RzmXt/cPuqNBvLHFxrXZvkm5FDBk3kiR5PeXSFDSlikDvRU8dfu+SnN3WzSAwPD/B6ZhQ2LeYg8J5R0VqqINqCAatIb4lYSS7a0QWXqL8MJ5eUjXbdnlGVFdk9RZpWgPtTVL99mEkmFP2T0T2StM7/w0/lgk1RvVJzBNaV1X85zGQ6qBq8XzK+IA/F8qvyfwgeKXDoZUhwQ38= 2021-05-27T13:36:14.588 INFO:teuthology.kill: smithi058.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzSGpQo//MuW20UrBCB5bXXWJiPvcxbQbK7Xyy916AInteTDmGjA3guUYQa5w6XbNWL5OKkKytaom91/f3zasM62MRKz2zrgR0Muzx7hJgKLfsvL6BgGrNWv+KyHYNlkP2nJlVYDERjO9zxSr4Xkk/2YyGzPusFCEcBWgILuPvtKA+Y6DPwKkzHaK37dRZiqcjB6gy5q8Ep+DImvfNGhGdM4pLy9aAYbaqV9dHChnUFQGrkjHz068NgS7Ep0tHGAQEi0qj000ZQ1rWbgUadLvdHRguVA6yEMDCWopk5sehMGBq0rIQBzMLo4hyG4eCJK1lujjrrgPbrB17+INyIMzdcJqs//v4AAH0YM98D5SJrmgKdGAVQ96EWVhgSK+Sq2omi1LntKvBNuXNOzpmL8MEt83k7hLO6FrxDTBZxJNQKgDB/mKnr7RObnR7TZMtFdxO+uvmYaRt3ICGQNqT5PLhxZyqwHmQyfYjwcjHti30sL1tbk/6wYiNA/KXV+ShET8= 2021-05-27T13:36:14.588 INFO:teuthology.kill:2021-05-27 13:36:14,587.587 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T13:36:14.589 INFO:teuthology.kill:2021-05-27 13:36:14,589.589 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T13:36:14.645 INFO:teuthology.kill:2021-05-27 13:36:14,644.644 INFO:teuthology.orchestra.console:Power off smithi028 2021-05-27T13:36:14.663 INFO:teuthology.kill:2021-05-27 13:36:14,662.662 INFO:teuthology.orchestra.console:Power off smithi058 2021-05-27T13:36:23.020 INFO:teuthology.kill:2021-05-27 13:36:23,020.020 INFO:teuthology.orchestra.console:Power off for smithi028 completed 2021-05-27T13:36:23.024 INFO:teuthology.kill:2021-05-27 13:36:23,024.024 INFO:teuthology.orchestra.console:Power off for smithi058 completed 2021-05-27T13:36:23.240 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2021-05-27T13:36:23.240 INFO:teuthology.task.internal:roles: ubuntu@smithi058.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2021-05-27T13:36:23.241 INFO:teuthology.misc:Compressing logs... 2021-05-27T13:36:23.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-05-27T13:37:23.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d92ec1407c30d60262a45a2844110a5c94d567cc/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-05-27T13:37:23.359 INFO:teuthology.kill:No teuthology processes running 2021-05-27T13:37:24.376 INFO:teuthology.kill:Nuking machines: ['smithi028', 'smithi058'] 2021-05-27T13:37:25.312 INFO:teuthology.kill:2021-05-27 13:37:25,312.312 INFO:teuthology.nuke:targets: 2021-05-27T13:37:25.313 INFO:teuthology.kill: smithi028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC38bO+jXR9ypoNHg7VM+sC4QALcqhWv8aMdzOPPAaB6M0hO+F5Ib7XnW/dIo+80bVyEQTliEjpsrjAa4WRFVSoBIeXXCgmr/SK7IMRoumYRE5Hv9qvXC47vULMfgBDlEdskw1UVJq/F2Zjj/PhOMVOE5AohfHSv5/lvO8izYzakYTMlXwoRI3jElDfI3bF+Y17k/YlF31/iGniebdX7rrZVd4QwWQrxr85JEYwmWTFDxyyuV9UXfdjcU/KBEB0FA5mPTYaTWGbNGu7ycpdjOnEwehs9whRigNXrLw+t6LfKrf56REh9RzmXt/cPuqNBvLHFxrXZvkm5FDBk3kiR5PeXSFDSlikDvRU8dfu+SnN3WzSAwPD/B6ZhQ2LeYg8J5R0VqqINqCAatIb4lYSS7a0QWXqL8MJ5eUjXbdnlGVFdk9RZpWgPtTVL99mEkmFP2T0T2StM7/w0/lgk1RvVJzBNaV1X85zGQ6qBq8XzK+IA/F8qvyfwgeKXDoZUhwQ38= 2021-05-27T13:37:25.314 INFO:teuthology.kill: smithi058.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzSGpQo//MuW20UrBCB5bXXWJiPvcxbQbK7Xyy916AInteTDmGjA3guUYQa5w6XbNWL5OKkKytaom91/f3zasM62MRKz2zrgR0Muzx7hJgKLfsvL6BgGrNWv+KyHYNlkP2nJlVYDERjO9zxSr4Xkk/2YyGzPusFCEcBWgILuPvtKA+Y6DPwKkzHaK37dRZiqcjB6gy5q8Ep+DImvfNGhGdM4pLy9aAYbaqV9dHChnUFQGrkjHz068NgS7Ep0tHGAQEi0qj000ZQ1rWbgUadLvdHRguVA6yEMDCWopk5sehMGBq0rIQBzMLo4hyG4eCJK1lujjrrgPbrB17+INyIMzdcJqs//v4AAH0YM98D5SJrmgKdGAVQ96EWVhgSK+Sq2omi1LntKvBNuXNOzpmL8MEt83k7hLO6FrxDTBZxJNQKgDB/mKnr7RObnR7TZMtFdxO+uvmYaRt3ICGQNqT5PLhxZyqwHmQyfYjwcjHti30sL1tbk/6wYiNA/KXV+ShET8= 2021-05-27T13:37:25.332 INFO:teuthology.kill:2021-05-27 13:37:25,331.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T13:37:25.334 INFO:teuthology.kill:2021-05-27 13:37:25,334.334 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-05-27T13:37:25.389 INFO:teuthology.kill:2021-05-27 13:37:25,389.389 INFO:teuthology.orchestra.console:Power off smithi058 2021-05-27T13:37:25.401 INFO:teuthology.kill:2021-05-27 13:37:25,401.401 INFO:teuthology.orchestra.console:Power off smithi028 2021-05-27T13:37:25.527 INFO:teuthology.kill:2021-05-27 13:37:25,527.527 INFO:teuthology.orchestra.console:Power off for smithi058 completed 2021-05-27T13:37:25.539 INFO:teuthology.kill:2021-05-27 13:37:25,539.539 INFO:teuthology.orchestra.console:Power off for smithi028 completed 2021-05-27T13:37:25.670 INFO:teuthology.kill:2021-05-27 13:37:25,669.669 INFO:teuthology.lock.ops:unlocked smithi058.front.sepia.ceph.com 2021-05-27T13:37:25.677 INFO:teuthology.kill:2021-05-27 13:37:25,676.676 INFO:teuthology.lock.ops:unlocked smithi028.front.sepia.ceph.com 2021-05-27T13:39:25.949 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15