2021-07-02T04:29:08.181 INFO:root:teuthology version: 1.1.0-f0b0ad0e 2021-07-02T04:29:08.281 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging 2021-07-02T04:29:08.283 INFO:teuthology.lock.ops:Updating [ubuntu@smithi134.front.sepia.ceph.com]: reset os type and version on server 2021-07-02T04:29:08.283 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2021-07-02T04:29:08.302 INFO:teuthology.lock.ops:Node 'ubuntu@smithi134.front.sepia.ceph.com' reimaging is complete 2021-07-02T04:29:08.303 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi137.front.sepia.ceph.com' reimaging 2021-07-02T04:29:08.303 INFO:teuthology.lock.ops:Updating [ubuntu@smithi137.front.sepia.ceph.com]: reset os type and version on server 2021-07-02T04:29:08.303 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2021-07-02T04:29:08.320 INFO:teuthology.lock.ops:Node 'ubuntu@smithi137.front.sepia.ceph.com' reimaging is complete 2021-07-02T04:29:08.531 INFO:teuthology.provision.fog.smithi134:Scheduling deploy of centos 8.3 2021-07-02T04:29:08.535 INFO:teuthology.provision.fog.smithi137:Scheduling deploy of centos 8.3 2021-07-02T04:29:08.933 INFO:teuthology.orchestra.console:Power off smithi134 2021-07-02T04:29:08.970 INFO:teuthology.orchestra.console:Power off smithi137 2021-07-02T04:29:09.081 INFO:teuthology.orchestra.console:Power off for smithi134 completed 2021-07-02T04:29:09.111 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2021-07-02T04:29:09.183 INFO:teuthology.orchestra.console:Power on smithi134 2021-07-02T04:29:09.215 INFO:teuthology.orchestra.console:Power on smithi137 2021-07-02T04:29:17.563 INFO:teuthology.orchestra.console:Power on for smithi134 completed 2021-07-02T04:29:17.639 INFO:teuthology.orchestra.console:Power on for smithi137 completed 2021-07-02T04:29:17.665 INFO:teuthology.provision.fog.smithi134:Waiting for deploy to finish 2021-07-02T04:29:17.741 INFO:teuthology.provision.fog.smithi137:Waiting for deploy to finish 2021-07-02T04:31:49.267 ERROR:teuthology.orchestra.connection:Error authenticating with smithi137.front.sepia.ceph.com: Authentication failed. 2021-07-02T04:33:30.017 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-07-02T04:33:50.126 INFO:teuthology.provision.fog.smithi137:Node is ready 2021-07-02T04:33:50.146 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137.front.sepia.ceph.com 2021-07-02T04:33:50.204 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.15.137 smithi137.front.sepia.ceph.com smithi137 2021-07-02T04:33:50.626 INFO:teuthology.provision.fog.smithi137:Deploy complete! 2021-07-02T04:34:53.269 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-07-02T04:34:53.579 INFO:teuthology.provision.fog.smithi134:Node is ready 2021-07-02T04:34:53.594 INFO:teuthology.orchestra.run.smithi134.stdout:smithi134.front.sepia.ceph.com 2021-07-02T04:34:53.649 INFO:teuthology.orchestra.run.smithi134.stdout:172.21.15.134 smithi134.front.sepia.ceph.com smithi134 2021-07-02T04:34:53.969 INFO:teuthology.provision.fog.smithi134:Deploy complete! 2021-07-02T04:34:54.093 INFO:teuthology.lock.ops:Checking smithi134.front.sepia.ceph.com 2021-07-02T04:34:54.094 INFO:teuthology.lock.ops:New key found. Updating... 2021-07-02T04:34:54.110 INFO:teuthology.lock.ops:Checking smithi137.front.sepia.ceph.com 2021-07-02T04:34:54.111 INFO:teuthology.lock.ops:New key found. Updating... 2021-07-02T04:34:54.139 INFO:teuthology.lock.ops:Updating [smithi134.front.sepia.ceph.com]: set os type and version on server 2021-07-02T04:34:54.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-02T04:34:54.355 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2021-07-02T04:34:54.412 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Linux" 2021-07-02T04:34:54.413 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="8" 2021-07-02T04:34:54.413 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2021-07-02T04:34:54.413 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2021-07-02T04:34:54.414 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="8" 2021-07-02T04:34:54.415 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el8" 2021-07-02T04:34:54.415 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Linux 8" 2021-07-02T04:34:54.416 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2021-07-02T04:34:54.416 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-07-02T04:34:54.417 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2021-07-02T04:34:54.417 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-07-02T04:34:54.417 INFO:teuthology.orchestra.run.smithi134.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-07-02T04:34:54.417 INFO:teuthology.orchestra.run.smithi134.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-07-02T04:34:54.419 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2021-07-02T04:34:54.434 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: set os type and version on server 2021-07-02T04:34:54.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-02T04:34:54.724 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2021-07-02T04:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Linux" 2021-07-02T04:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8" 2021-07-02T04:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2021-07-02T04:34:54.779 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2021-07-02T04:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8" 2021-07-02T04:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2021-07-02T04:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Linux 8" 2021-07-02T04:34:54.780 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2021-07-02T04:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-07-02T04:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2021-07-02T04:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-07-02T04:34:54.781 INFO:teuthology.orchestra.run.smithi137.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-07-02T04:34:54.782 INFO:teuthology.orchestra.run.smithi137.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-07-02T04:34:54.782 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2021-07-02T04:34:54.874 INFO:teuthology.dispatcher.supervisor:Running job 6248689 2021-07-02T04:34:54.874 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/virtualenv/bin/teuthology -v --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-07-02_03:53:50-rados-wip-kefu-testing-2021-07-02-0958-distro-basic-smithi/6248689 --name kchai-2021-07-02_03:53:50-rados-wip-kefu-testing-2021-07-02-0958-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects} -- /home/teuthworker/archive/kchai-2021-07-02_03:53:50-rados-wip-kefu-testing-2021-07-02-0958-distro-basic-smithi/6248689/orig.config.yaml 2021-07-02T04:34:54.880 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/kchai-2021-07-02_03:53:50-rados-wip-kefu-testing-2021-07-02-0958-distro-basic-smithi/6248689 2021-07-02T04:34:54.881 INFO:teuthology.dispatcher.supervisor:Job PID: 20462 2021-07-02T04:34:54.881 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-07-02T16:35:13.077 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-07-02T16:35:13.523 INFO:teuthology.kill:Killing Pids: {20462} 2021-07-02T16:35:19.573 INFO:teuthology.kill:Nuking machines: ['smithi134', 'smithi137'] 2021-07-02T16:35:21.470 INFO:teuthology.kill:2021-07-02 16:35:21,469.469 INFO:teuthology.nuke:targets: 2021-07-02T16:35:21.471 INFO:teuthology.kill: smithi134.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7POTBLcSMAC9Uyd5NiIgHGWRnrhd88EnwE5I+Viplwiw/7ZyExm9kd2yODY99rL4mw4Fsfhm7Yz9lfP/2+XIzE+Ft8ElxWVwPUIQ+iaDMTRKboLvSrqz6rA5Kjt4Fcp172s+S1cuHSD1N3i/WyuUiVphYX8+a3OxmuNh4T2bdmZap7FJGvhBu86h6SYq8syaVOgqOfttUAFQRf922xTF+VdMZwk2lOUs26I/hjLgQka4rwFPR+P3KWOvAPqd0Owe+l1deBbUMbz/oNUI4tlEgKhlWl0KmJtZOrK0wZoeUP4Sj4CpHVXpMarAiyqZtQyF7Unfs4hbsM7YuxH1b3lpXJOv3EdiROOhNAuf6qwm57uAj1slq7hlmXbRv0JVrvdAq1d8HeVwz/+v7gygJtbJ03FlLrxJC+JrenCC0ktV5LBrp0Kk0/TWUgggMWZIj9jGjjXjuuPjNoyyUbbF/yri5D/Q1/po7gYPt3D3tALFcyzFf7XL8AaKHIDHsgiiqKtc= 2021-07-02T16:35:21.471 INFO:teuthology.kill: smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoRf1J7KKuEU9n2t6k5EE4GhoEZ3xyBCePCAskJCynqQWvWYUhpBRG/8f5kMazTUczVlb3lGCbGUfF3H1gmdgC9vWFK6SyaVDxhlrSBDumsMh3O44FJNc5alSULAB092bSudiqHrJHSSnSemIJRNTrvumtOMdKe2FaO9DuujuYA1eUOXT1NZUAXLMAIGhNfPU6ikPElXk04ub3PrcwGc966mgayeUpG7YaaLcOkgOZ0Xh8d22hnKrA85Wbif/0YRDVpGgr4NYWDesUtLbEWakyNlVq+h6ZCIwxSN9fMNWz/StW2F5SBPSSCHgcWl0S2SXRgbSb1EDDjLQ8H0h7EuvUxnLqMhzSpmRW0kCJgBEev0xkeZ+6G8z0XWUiTuYB73O5aeypTt9WM+alHa2BMvwyvyP+6yx3mQFR/uGJq/3ZLyBP0WpKFPhWggK6h8gYjgp5u8cBLxTIjNFkgMJGA5154fjC5k+fOgKyUL8sAuaejN8bhVxReXQ+rHxroPeNu58= 2021-07-02T16:35:21.471 INFO:teuthology.kill:2021-07-02 16:35:21,469.469 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-02T16:35:21.484 INFO:teuthology.kill:2021-07-02 16:35:21,482.482 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-02T16:35:21.618 INFO:teuthology.kill:2021-07-02 16:35:21,616.616 INFO:teuthology.orchestra.console:Power off smithi134 2021-07-02T16:35:21.659 INFO:teuthology.kill:2021-07-02 16:35:21,649.649 INFO:teuthology.orchestra.console:Power off smithi137 2021-07-02T16:35:30.044 INFO:teuthology.kill:2021-07-02 16:35:30,043.043 INFO:teuthology.orchestra.console:Power off for smithi134 completed 2021-07-02T16:35:30.051 INFO:teuthology.kill:2021-07-02 16:35:30,051.051 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2021-07-02T16:35:30.427 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-07-02T16:35:30.429 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-07-02T16:35:30.429 INFO:teuthology.misc:Compressing logs... 2021-07-02T16:35:30.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-07-02T16:36:04.943 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/teuthology/misc.py", line 1378, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/teuthology/orchestra/remote.py", line 509, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f0b0ad0edc5c0086ebe8886a9d8720792d57839f/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-07-02T16:36:05.157 INFO:teuthology.kill:No teuthology processes running 2021-07-02T16:36:07.093 INFO:teuthology.kill:Nuking machines: ['smithi134', 'smithi137'] 2021-07-02T16:36:09.516 INFO:teuthology.kill:2021-07-02 16:36:09,515.515 INFO:teuthology.nuke:targets: 2021-07-02T16:36:09.517 INFO:teuthology.kill: smithi134.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7POTBLcSMAC9Uyd5NiIgHGWRnrhd88EnwE5I+Viplwiw/7ZyExm9kd2yODY99rL4mw4Fsfhm7Yz9lfP/2+XIzE+Ft8ElxWVwPUIQ+iaDMTRKboLvSrqz6rA5Kjt4Fcp172s+S1cuHSD1N3i/WyuUiVphYX8+a3OxmuNh4T2bdmZap7FJGvhBu86h6SYq8syaVOgqOfttUAFQRf922xTF+VdMZwk2lOUs26I/hjLgQka4rwFPR+P3KWOvAPqd0Owe+l1deBbUMbz/oNUI4tlEgKhlWl0KmJtZOrK0wZoeUP4Sj4CpHVXpMarAiyqZtQyF7Unfs4hbsM7YuxH1b3lpXJOv3EdiROOhNAuf6qwm57uAj1slq7hlmXbRv0JVrvdAq1d8HeVwz/+v7gygJtbJ03FlLrxJC+JrenCC0ktV5LBrp0Kk0/TWUgggMWZIj9jGjjXjuuPjNoyyUbbF/yri5D/Q1/po7gYPt3D3tALFcyzFf7XL8AaKHIDHsgiiqKtc= 2021-07-02T16:36:09.517 INFO:teuthology.kill: smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoRf1J7KKuEU9n2t6k5EE4GhoEZ3xyBCePCAskJCynqQWvWYUhpBRG/8f5kMazTUczVlb3lGCbGUfF3H1gmdgC9vWFK6SyaVDxhlrSBDumsMh3O44FJNc5alSULAB092bSudiqHrJHSSnSemIJRNTrvumtOMdKe2FaO9DuujuYA1eUOXT1NZUAXLMAIGhNfPU6ikPElXk04ub3PrcwGc966mgayeUpG7YaaLcOkgOZ0Xh8d22hnKrA85Wbif/0YRDVpGgr4NYWDesUtLbEWakyNlVq+h6ZCIwxSN9fMNWz/StW2F5SBPSSCHgcWl0S2SXRgbSb1EDDjLQ8H0h7EuvUxnLqMhzSpmRW0kCJgBEev0xkeZ+6G8z0XWUiTuYB73O5aeypTt9WM+alHa2BMvwyvyP+6yx3mQFR/uGJq/3ZLyBP0WpKFPhWggK6h8gYjgp5u8cBLxTIjNFkgMJGA5154fjC5k+fOgKyUL8sAuaejN8bhVxReXQ+rHxroPeNu58= 2021-07-02T16:36:09.592 INFO:teuthology.kill:2021-07-02 16:36:09,591.591 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-02T16:36:09.593 INFO:teuthology.kill:2021-07-02 16:36:09,592.592 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-07-02T16:36:09.772 INFO:teuthology.kill:2021-07-02 16:36:09,770.770 INFO:teuthology.orchestra.console:Power off smithi137 2021-07-02T16:36:09.844 INFO:teuthology.kill:2021-07-02 16:36:09,843.843 INFO:teuthology.orchestra.console:Power off smithi134 2021-07-02T16:36:10.061 INFO:teuthology.kill:2021-07-02 16:36:10,051.051 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2021-07-02T16:36:10.125 INFO:teuthology.kill:2021-07-02 16:36:10,101.101 INFO:teuthology.orchestra.console:Power off for smithi134 completed 2021-07-02T16:36:10.252 INFO:teuthology.kill:2021-07-02 16:36:10,247.247 INFO:teuthology.lock.ops:unlocked smithi137.front.sepia.ceph.com 2021-07-02T16:36:10.291 INFO:teuthology.kill:2021-07-02 16:36:10,289.289 INFO:teuthology.lock.ops:unlocked smithi134.front.sepia.ceph.com 2021-07-02T16:38:10.787 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-07-02T16:38:10.920 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-07-02T16:38:10.921 INFO:teuthology.nuke:Checking targets against current locks 2021-07-02T16:38:11.000 INFO:teuthology.nuke:Not nuking smithi137.front.sepia.ceph.com because description doesn't match 2021-07-02T16:38:11.001 INFO:teuthology.nuke:Not nuking smithi134.front.sepia.ceph.com because description doesn't match