2022-03-13T03:38:38.448 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-13T03:38:38.574 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging 2022-03-13T03:38:38.576 INFO:teuthology.lock.ops:Updating [ubuntu@smithi183.front.sepia.ceph.com]: reset os type and version on server 2022-03-13T03:38:38.576 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2022-03-13T03:38:38.597 INFO:teuthology.lock.ops:Node 'ubuntu@smithi183.front.sepia.ceph.com' reimaging is complete 2022-03-13T03:38:38.598 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging 2022-03-13T03:38:38.598 INFO:teuthology.lock.ops:Updating [ubuntu@smithi195.front.sepia.ceph.com]: reset os type and version on server 2022-03-13T03:38:38.598 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2022-03-13T03:38:38.618 INFO:teuthology.lock.ops:Node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging is complete 2022-03-13T03:38:38.814 INFO:teuthology.provision.fog.smithi183:Scheduling deploy of ubuntu 18.04 2022-03-13T03:38:38.833 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of ubuntu 18.04 2022-03-13T03:38:39.165 INFO:teuthology.orchestra.console:Power off smithi183 2022-03-13T03:38:39.178 INFO:teuthology.orchestra.console:Power off smithi195 2022-03-13T03:38:47.539 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2022-03-13T03:38:47.640 INFO:teuthology.orchestra.console:Power on smithi195 2022-03-13T03:38:51.644 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2022-03-13T03:38:51.745 INFO:teuthology.orchestra.console:Power on smithi183 2022-03-13T03:38:56.001 INFO:teuthology.orchestra.console:Power on for smithi195 completed 2022-03-13T03:38:56.102 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish 2022-03-13T03:39:00.109 INFO:teuthology.orchestra.console:Power on for smithi183 completed 2022-03-13T03:39:00.210 INFO:teuthology.provision.fog.smithi183:Waiting for deploy to finish 2022-03-13T03:43:21.295 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:26.153 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:27.309 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:32.169 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:33.322 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:38.183 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:39.335 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:44.197 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:45.349 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2022-03-13T03:43:50.305 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-13T03:43:50.531 INFO:teuthology.provision.fog.smithi195:Node is ready 2022-03-13T03:43:50.581 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com 2022-03-13T03:43:50.631 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195 2022-03-13T03:43:50.938 INFO:teuthology.provision.fog.smithi195:Deploy complete! 2022-03-13T03:43:51.438 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-13T03:43:51.712 INFO:teuthology.provision.fog.smithi183:Node is ready 2022-03-13T03:43:51.760 INFO:teuthology.orchestra.run.smithi183.stdout:smithi183.front.sepia.ceph.com 2022-03-13T03:43:51.809 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.15.183 smithi183.front.sepia.ceph.com smithi183 2022-03-13T03:43:52.101 INFO:teuthology.provision.fog.smithi183:Deploy complete! 2022-03-13T03:43:52.264 INFO:teuthology.lock.ops:Checking smithi183.front.sepia.ceph.com 2022-03-13T03:43:52.265 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-13T03:43:52.285 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com 2022-03-13T03:43:52.285 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-13T03:43:52.307 INFO:teuthology.lock.ops:Updating [smithi183.front.sepia.ceph.com]: set os type and version on server 2022-03-13T03:43:52.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2022-03-13T03:43:52.518 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2022-03-13T03:43:52.568 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="Ubuntu" 2022-03-13T03:43:52.569 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-03-13T03:43:52.569 INFO:teuthology.orchestra.run.smithi183.stdout:ID=ubuntu 2022-03-13T03:43:52.569 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE=debian 2022-03-13T03:43:52.569 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-03-13T03:43:52.569 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="18.04" 2022-03-13T03:43:52.570 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-13T03:43:52.570 INFO:teuthology.orchestra.run.smithi183.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-13T03:43:52.570 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-13T03:43:52.570 INFO:teuthology.orchestra.run.smithi183.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-13T03:43:52.571 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_CODENAME=bionic 2022-03-13T03:43:52.571 INFO:teuthology.orchestra.run.smithi183.stdout:UBUNTU_CODENAME=bionic 2022-03-13T03:43:52.572 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2022-03-13T03:43:52.591 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server 2022-03-13T03:43:52.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-03-13T03:43:52.893 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2022-03-13T03:43:52.942 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="Ubuntu" 2022-03-13T03:43:52.943 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="18.04.6 LTS (Bionic Beaver)" 2022-03-13T03:43:52.943 INFO:teuthology.orchestra.run.smithi195.stdout:ID=ubuntu 2022-03-13T03:43:52.943 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE=debian 2022-03-13T03:43:52.943 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="Ubuntu 18.04.6 LTS" 2022-03-13T03:43:52.943 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="18.04" 2022-03-13T03:43:52.944 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://www.ubuntu.com/" 2022-03-13T03:43:52.944 INFO:teuthology.orchestra.run.smithi195.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-03-13T03:43:52.944 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-03-13T03:43:52.944 INFO:teuthology.orchestra.run.smithi195.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-03-13T03:43:52.945 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_CODENAME=bionic 2022-03-13T03:43:52.945 INFO:teuthology.orchestra.run.smithi195.stdout:UBUNTU_CODENAME=bionic 2022-03-13T03:43:52.945 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2022-03-13T03:43:53.041 INFO:teuthology.dispatcher.supervisor:Running job 6733991 2022-03-13T03:43:53.042 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_fbfc7b381c1d662565abb9d3552d468766bf123c/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6733991 --name teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi --description rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} -- /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6733991/orig.config.yaml 2022-03-13T03:43:53.046 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-03-13_03:31:03-rados-pacific-distro-default-smithi/6733991 2022-03-13T03:43:53.046 INFO:teuthology.dispatcher.supervisor:Job PID: 13346 2022-03-13T03:43:53.047 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-13T10:14:22.676 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-13T10:14:22.872 INFO:teuthology.kill:Killing Pids: {13346} 2022-03-13T10:14:24.197 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['host.a', 'client.0'] 2022-03-13T10:14:24.198 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['host.b', 'client.1'] 2022-03-13T10:14:24.199 INFO:teuthology.misc:Compressing logs... 2022-03-13T10:14:24.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2022-03-13T10:14:24.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-03-13T10:14:24.490 INFO:teuthology.orchestra.run.smithi183.stderr:find: '/var/lib/ceph/crash': No such file or directory 2022-03-13T10:14:24.808 INFO:teuthology.orchestra.run.smithi195.stderr:find: '/var/lib/ceph/crash': No such file or directory 2022-03-13T10:14:24.826 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-03-13T10:14:24.826 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2022-03-13T10:14:24.865 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2022-03-13T10:14:24.866 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2022-03-13T10:14:24.868 INFO:teuthology.misc:Compressing logs... 2022-03-13T10:14:25.141 INFO:teuthology.misc:Compressing logs... 2022-03-13T10:14:27.295 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-03-13T10:14:27.321 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-03-13T10:14:29.037 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-03-13T10:14:29.118 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-03-13T10:14:32.654 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-mon.smithi195.log: file size changed while zipping 2022-03-13T10:14:34.473 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-03-13T10:14:36.260 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-03-13T10:14:38.114 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-mon.smithi183.log: file size changed while zipping 2022-03-13T10:14:39.961 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-mgr.smithi183.bxxfov.log: file size changed while zipping 2022-03-13T10:14:41.944 INFO:teuthology.orchestra.run.smithi183.stderr:gzip: /var/log/ceph/b5ef1db8-a280-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-03-13T10:14:47.935 INFO:teuthology.kill:No teuthology processes running 2022-03-13T10:14:49.195 INFO:teuthology.kill:Nuking machines: ['smithi183', 'smithi195'] 2022-03-13T10:14:50.232 INFO:teuthology.kill:2022-03-13 10:14:50,231.231 INFO:teuthology.nuke:targets: 2022-03-13T10:14:50.233 INFO:teuthology.kill: smithi183.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKM1rQfHWT+VaGfw4NKFZ73ZG2T/1CfpcuvUKlTueqnl9Cdt8zmpy1DYQJ61trtxgCP2nMjk/n+s/h2ttu4W/Xz0yaTE15EPCQ51FGPLOo+hJXwwFUWcdaBiOSgYZFJXEjJ9OAQOkDF83QdZtsx9aLMRo5H7SPqLqLPfgxLS5i9MjgP9P5kjx03XBnVPKMM7GsU0fuiGC7M6XjOjljIn0HzuL4DCMCiEIkvf6umHqfufX7Rb5uuM9vSLpZmrWErMZbDmURmp8Bvvxvu87wl6zOsuAHdz4BSyqMcMu1jlbo4juz3k+/GSuanTWVUN1cSsAff6oSE5FYN8pY1SQZx8Kn 2022-03-13T10:14:50.233 INFO:teuthology.kill: smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC69ogSUkYkXY2ziltQGsojYI/JRrQuzBij4hYPJzUF9UeT8cpqhmgQEcugzrOeuWTbrT8lyPlmC4ckYZ/jk7AKFwfvzD8G9Ke7wonn+dNFDXAf+g7hQiBWSstl9yR34fYB8eolSatKf9kAzpBB6KOpGHtK1yIk/OC0/Uwkoeil7nzDJ4uIn3VwuHoihkzkww1e0uikyd54avKikVWUMX3kWARXzmMJSJ0FBNKTIQOmfYfJw0BuN9Cx1/npgjft5FotqB+effA0dK7oKO3MwoSrr/c/+i1CDhyuDe8vTVzMJzTVhe8MIAGhr9Z3rOepLoSVgOVqkq4VYj/lUp82Yvon 2022-03-13T10:14:50.258 INFO:teuthology.kill:2022-03-13 10:14:50,258.258 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-13T10:14:50.262 INFO:teuthology.kill:2022-03-13 10:14:50,262.262 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-13T10:14:50.350 INFO:teuthology.kill:2022-03-13 10:14:50,348.348 INFO:teuthology.orchestra.console:Power off smithi195 2022-03-13T10:14:50.370 INFO:teuthology.kill:2022-03-13 10:14:50,369.369 INFO:teuthology.orchestra.console:Power off smithi183 2022-03-13T10:14:58.742 INFO:teuthology.kill:2022-03-13 10:14:58,741.741 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2022-03-13T10:14:58.753 INFO:teuthology.kill:2022-03-13 10:14:58,752.752 INFO:teuthology.orchestra.console:Power off for smithi183 completed 2022-03-13T10:14:58.893 INFO:teuthology.kill:2022-03-13 10:14:58,893.893 INFO:teuthology.lock.ops:unlocked: smithi195.front.sepia.ceph.com 2022-03-13T10:14:58.900 INFO:teuthology.kill:2022-03-13 10:14:58,900.900 INFO:teuthology.lock.ops:unlocked: smithi183.front.sepia.ceph.com 2022-03-13T10:16:59.255 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-13T10:16:59.351 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi183 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-13_07:01:03-rados-master-distro-default-smithi/6734660 2022-03-13T10:16:59.352 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi195 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-13_07:01:03-rados-master-distro-default-smithi/6734660