2023-01-15T12:43:46.357 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2023-01-15T12:43:46.506 INFO:teuthology.lock.ops:Start node 'smithi019.front.sepia.ceph.com' reimaging
2023-01-15T12:43:46.506 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: reset os type and version on server
2023-01-15T12:43:46.506 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server
2023-01-15T12:43:46.522 INFO:teuthology.lock.ops:Node 'smithi019.front.sepia.ceph.com' reimaging is complete
2023-01-15T12:43:46.522 INFO:teuthology.lock.ops:Start node 'smithi142.front.sepia.ceph.com' reimaging
2023-01-15T12:43:46.522 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: reset os type and version on server
2023-01-15T12:43:46.522 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2023-01-15T12:43:46.539 INFO:teuthology.lock.ops:Node 'smithi142.front.sepia.ceph.com' reimaging is complete
2023-01-15T12:43:46.540 INFO:teuthology.lock.ops:Start node 'smithi146.front.sepia.ceph.com' reimaging
2023-01-15T12:43:46.540 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: reset os type and version on server
2023-01-15T12:43:46.540 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2023-01-15T12:43:46.555 INFO:teuthology.lock.ops:Node 'smithi146.front.sepia.ceph.com' reimaging is complete
2023-01-15T12:43:46.674 INFO:teuthology.provision.fog.smithi019:Scheduling deploy of ubuntu 20.04
2023-01-15T12:43:46.687 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of ubuntu 20.04
2023-01-15T12:43:46.708 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of ubuntu 20.04
2023-01-15T12:43:47.117 INFO:teuthology.orchestra.console:Power off smithi019
2023-01-15T12:43:47.129 INFO:teuthology.orchestra.console:Power off smithi142
2023-01-15T12:43:47.139 INFO:teuthology.orchestra.console:Power off smithi146
2023-01-15T12:43:47.266 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2023-01-15T12:43:47.271 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2023-01-15T12:43:47.276 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2023-01-15T12:43:47.367 INFO:teuthology.orchestra.console:Power on smithi019
2023-01-15T12:43:47.375 INFO:teuthology.orchestra.console:Power on smithi142
2023-01-15T12:43:47.382 INFO:teuthology.orchestra.console:Power on smithi146
2023-01-15T12:43:51.625 INFO:teuthology.orchestra.console:Power on for smithi019 completed
2023-01-15T12:43:51.643 INFO:teuthology.orchestra.console:Power on for smithi146 completed
2023-01-15T12:43:51.726 INFO:teuthology.provision.fog.smithi019:Waiting for deploy to finish
2023-01-15T12:43:51.744 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish
2023-01-15T12:43:55.744 INFO:teuthology.orchestra.console:Power on for smithi142 completed
2023-01-15T12:43:55.844 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish
2023-01-15T12:47:17.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-01-15T12:47:26.212 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-01-15T12:47:35.268 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-01-15T12:47:44.328 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-01-15T12:47:53.412 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.19
2023-01-15T12:48:59.425 WARNING:teuthology.provision.fog:timed out
2023-01-15T12:49:05.653 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-15T12:49:06.152 INFO:teuthology.provision.fog.smithi019:Node is ready
2023-01-15T12:49:06.204 INFO:teuthology.orchestra.run.smithi019.stdout:smithi019.front.sepia.ceph.com
2023-01-15T12:49:06.211 INFO:teuthology.orchestra.run.smithi019.stdout:172.21.15.19 smithi019.front.sepia.ceph.com smithi019
2023-01-15T12:49:06.391 INFO:teuthology.provision.fog.smithi019:Deploy complete!
2023-01-15T12:55:11.945 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-01-15T12:55:21.032 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-01-15T12:55:30.119 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-01-15T12:55:39.204 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-01-15T12:55:48.265 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-01-15T12:55:57.348 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-01-15T12:57:03.367 WARNING:teuthology.provision.fog:timed out
2023-01-15T12:57:09.572 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-15T12:57:10.090 INFO:teuthology.provision.fog.smithi142:Node is ready
2023-01-15T12:57:10.097 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com
2023-01-15T12:57:10.145 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142
2023-01-15T12:57:10.391 INFO:teuthology.provision.fog.smithi142:Deploy complete!
2023-01-15T12:59:10.698 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T12:59:19.753 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T12:59:28.836 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T12:59:37.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T12:59:46.985 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T12:59:56.071 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T13:00:05.129 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T13:00:14.222 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146
2023-01-15T13:00:52.480 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-15T13:01:13.721 INFO:teuthology.provision.fog.smithi146:Node is ready
2023-01-15T13:01:13.739 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com
2023-01-15T13:01:13.793 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146
2023-01-15T13:01:14.095 INFO:teuthology.provision.fog.smithi146:Deploy complete!
2023-01-15T13:01:14.965 INFO:teuthology.lock.ops:Checking smithi019.front.sepia.ceph.com
2023-01-15T13:01:14.965 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-15T13:01:15.022 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com
2023-01-15T13:01:15.023 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-15T13:01:15.085 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com
2023-01-15T13:01:15.085 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-15T13:01:15.172 INFO:teuthology.lock.ops:Updating [smithi019.front.sepia.ceph.com]: set os type and version on server
2023-01-15T13:01:15.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-15T13:01:15.562 INFO:teuthology.orchestra.run.smithi019.stdout:x86_64
2023-01-15T13:01:15.618 INFO:teuthology.orchestra.run.smithi019.stdout:NAME="Ubuntu"
2023-01-15T13:01:15.618 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-01-15T13:01:15.618 INFO:teuthology.orchestra.run.smithi019.stdout:ID=ubuntu
2023-01-15T13:01:15.618 INFO:teuthology.orchestra.run.smithi019.stdout:ID_LIKE=debian
2023-01-15T13:01:15.618 INFO:teuthology.orchestra.run.smithi019.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-01-15T13:01:15.642 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_ID="20.04"
2023-01-15T13:01:15.642 INFO:teuthology.orchestra.run.smithi019.stdout:HOME_URL="https://www.ubuntu.com/"
2023-01-15T13:01:15.642 INFO:teuthology.orchestra.run.smithi019.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-01-15T13:01:15.642 INFO:teuthology.orchestra.run.smithi019.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-01-15T13:01:15.642 INFO:teuthology.orchestra.run.smithi019.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-01-15T13:01:15.643 INFO:teuthology.orchestra.run.smithi019.stdout:VERSION_CODENAME=focal
2023-01-15T13:01:15.643 INFO:teuthology.orchestra.run.smithi019.stdout:UBUNTU_CODENAME=focal
2023-01-15T13:01:15.646 INFO:teuthology.lock.ops:Updating smithi019.front.sepia.ceph.com on lock server
2023-01-15T13:01:15.759 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server
2023-01-15T13:01:15.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-15T13:01:16.056 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64
2023-01-15T13:01:16.068 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Ubuntu"
2023-01-15T13:01:16.068 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-01-15T13:01:16.068 INFO:teuthology.orchestra.run.smithi142.stdout:ID=ubuntu
2023-01-15T13:01:16.068 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE=debian
2023-01-15T13:01:16.068 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-01-15T13:01:16.068 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="20.04"
2023-01-15T13:01:16.068 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.ubuntu.com/"
2023-01-15T13:01:16.069 INFO:teuthology.orchestra.run.smithi142.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-01-15T13:01:16.069 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-01-15T13:01:16.069 INFO:teuthology.orchestra.run.smithi142.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-01-15T13:01:16.069 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_CODENAME=focal
2023-01-15T13:01:16.069 INFO:teuthology.orchestra.run.smithi142.stdout:UBUNTU_CODENAME=focal
2023-01-15T13:01:16.069 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2023-01-15T13:01:16.118 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server
2023-01-15T13:01:16.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-15T13:01:16.458 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64
2023-01-15T13:01:16.506 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu"
2023-01-15T13:01:16.506 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-01-15T13:01:16.506 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu
2023-01-15T13:01:16.506 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian
2023-01-15T13:01:16.507 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-01-15T13:01:16.507 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="20.04"
2023-01-15T13:01:16.512 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/"
2023-01-15T13:01:16.513 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-01-15T13:01:16.513 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-01-15T13:01:16.513 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-01-15T13:01:16.513 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=focal
2023-01-15T13:01:16.513 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=focal
2023-01-15T13:01:16.528 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server
2023-01-15T13:01:16.814 INFO:teuthology.dispatcher.supervisor:Running job 7142573
2023-01-15T13:01:16.814 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2023-01-15_05:00:02-smoke-main-distro-default-smithi/7142573 --name teuthology-2023-01-15_05:00:02-smoke-main-distro-default-smithi --description smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_bench}} -- /home/teuthworker/archive/teuthology-2023-01-15_05:00:02-smoke-main-distro-default-smithi/7142573/orig.config.yaml
2023-01-15T13:01:16.843 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2023-01-15_05:00:02-smoke-main-distro-default-smithi/7142573
2023-01-15T13:01:16.843 INFO:teuthology.dispatcher.supervisor:Job PID: 17044
2023-01-15T13:01:16.843 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-01-16T01:02:46.237 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:02:46.566 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:02:47.359 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:02:47.360 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:02:47.360 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:02:47.360 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:02:47.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:02:47.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:02:47.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:02:47.978 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:02:48.126 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:02:58.025 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping
2023-01-16T01:03:03.240 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping
2023-01-16T01:03:11.136 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping
2023-01-16T01:03:11.531 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping
2023-01-16T01:03:13.109 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping
2023-01-16T01:03:13.345 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping
2023-01-16T01:03:15.344 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping
2023-01-16T01:03:15.419 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping
2023-01-16T01:03:17.150 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping
2023-01-16T01:05:07.065 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:05:07.818 INFO:teuthology.kill:Nuking machines: ['smithi019', 'smithi142', 'smithi146']
2023-01-16T01:05:08.413 INFO:teuthology.kill:2023-01-16 01:05:08,412.412 INFO:teuthology.nuke:targets:
2023-01-16T01:05:08.413 INFO:teuthology.kill:  smithi019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0=
2023-01-16T01:05:08.413 INFO:teuthology.kill:  smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0=
2023-01-16T01:05:08.413 INFO:teuthology.kill:  smithi146.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0=
2023-01-16T01:05:08.414 INFO:teuthology.kill:2023-01-16 01:05:08,413.413 INFO:teuthology.nuke:Checking targets against current locks
2023-01-16T01:05:08.465 INFO:teuthology.kill:2023-01-16 01:05:08,465.465 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-16T01:05:08.488 INFO:teuthology.kill:2023-01-16 01:05:08,487.487 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-16T01:05:08.509 INFO:teuthology.kill:2023-01-16 01:05:08,509.509 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-16T01:05:08.541 INFO:teuthology.kill:2023-01-16 01:05:08,541.541 INFO:teuthology.orchestra.console:Power off smithi019
2023-01-16T01:05:08.580 INFO:teuthology.kill:2023-01-16 01:05:08,580.580 INFO:teuthology.orchestra.console:Power off smithi146
2023-01-16T01:05:08.591 INFO:teuthology.kill:2023-01-16 01:05:08,591.591 INFO:teuthology.orchestra.console:Power off smithi142
2023-01-16T01:05:16.928 INFO:teuthology.kill:2023-01-16 01:05:16,928.928 INFO:teuthology.orchestra.console:Power off for smithi019 completed
2023-01-16T01:05:16.949 INFO:teuthology.kill:2023-01-16 01:05:16,949.949 INFO:teuthology.orchestra.console:Power off for smithi146 completed
2023-01-16T01:05:16.968 INFO:teuthology.kill:2023-01-16 01:05:16,967.967 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2023-01-16T01:05:17.101 INFO:teuthology.kill:2023-01-16 01:05:17,101.101 INFO:teuthology.lock.ops:unlocked: smithi019.front.sepia.ceph.com
2023-01-16T01:05:17.103 INFO:teuthology.kill:2023-01-16 01:05:17,102.102 INFO:teuthology.lock.ops:unlocked: smithi146.front.sepia.ceph.com
2023-01-16T01:05:17.121 INFO:teuthology.kill:2023-01-16 01:05:17,121.121 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com
2023-01-16T01:07:17.352 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:07:17.402 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:07:18.123 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:07:18.123 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:07:18.124 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:07:18.124 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:07:18.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:07:36.485 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T01:07:36.538 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:07:37.195 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:09:37.346 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:09:37.400 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:09:38.173 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:09:38.173 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:09:38.173 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:09:38.173 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:09:38.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:09:41.257 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T01:09:41.306 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:09:42.009 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:11:42.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:11:42.215 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:11:42.907 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:11:42.908 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:11:42.908 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:11:42.908 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:11:42.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:11:45.989 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T01:11:46.038 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:11:46.786 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:13:46.934 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:13:47.031 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:13:47.836 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:13:47.837 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:13:47.837 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:13:47.837 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:13:47.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:14:04.135 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T01:14:04.185 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:14:04.882 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:16:05.027 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:16:05.081 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:16:05.826 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:16:05.827 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:16:05.827 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:16:05.827 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:16:05.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:16:37.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:16:37.704 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:16:37.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:16:37.853 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:16:38.058 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:16:38.094 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:16:38.095 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:16:38.147 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:16:38.148 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:16:38.211 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:16:38.211 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:16:38.214 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:16:38.241 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T01:16:38.316 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T01:16:38.317 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:16:38.448 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:16:38.476 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T01:16:38.492 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T01:16:38.524 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T01:16:38.554 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T01:16:38.554 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:16:38.595 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T01:16:38.596 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:16:38.638 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T01:16:38.638 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:16:38.688 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:16:39.406 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:18:39.574 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:18:39.623 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:18:40.362 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:18:40.362 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:18:40.362 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:18:40.362 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:18:40.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:18:40.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:18:40.566 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:18:40.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:18:40.797 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:18:41.017 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:18:41.047 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:18:41.048 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:18:41.093 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:18:41.093 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:18:41.138 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:18:41.138 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:18:41.141 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:18:41.167 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T01:18:41.252 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T01:18:41.252 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:18:41.387 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:18:41.414 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T01:18:41.431 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T01:18:41.464 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T01:18:41.494 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T01:18:41.494 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:18:41.535 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T01:18:41.535 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:18:41.576 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T01:18:41.577 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:18:41.627 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:18:42.308 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:20:42.497 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:20:42.545 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:20:43.266 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:20:43.266 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:20:43.266 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:20:43.266 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:20:43.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:21:00.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:21:00.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:21:00.535 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:21:00.558 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:21:00.771 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:21:00.879 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:21:00.879 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:21:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:21:00.924 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:21:00.968 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:21:00.968 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:21:00.970 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:21:01.017 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:21:01.053 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:21:01.068 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T01:21:01.077 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T01:21:01.121 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:21:01.804 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:23:01.964 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:23:02.013 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:23:02.739 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:23:02.740 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:23:02.740 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:23:02.740 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:23:02.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:23:02.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:23:02.985 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:23:03.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:23:03.185 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:23:03.401 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:23:03.446 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:23:03.446 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:23:03.488 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:23:03.488 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:23:03.530 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:23:03.530 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:23:03.532 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:23:03.577 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:23:03.586 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T01:23:03.635 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:23:04.336 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:25:04.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:25:04.536 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:25:05.249 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:25:05.249 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:25:05.250 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:25:05.250 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:25:05.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:25:05.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:25:05.504 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:25:05.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:25:05.982 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:25:05.983 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:25:06.077 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:25:06.122 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:25:06.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T01:25:06.176 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:25:06.884 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:27:06.942 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:27:06.996 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:27:07.746 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:27:07.747 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:27:07.747 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:27:07.747 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:27:07.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:27:07.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:27:08.002 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:27:08.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:27:08.467 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:27:08.467 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:27:08.560 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:27:08.605 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:27:08.606 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:27:08.608 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T01:27:08.657 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:27:09.459 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:29:09.604 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:29:09.649 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:29:10.365 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:29:10.366 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:29:10.366 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:29:10.366 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:29:10.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:29:10.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:29:10.612 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:29:45.124 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:29:45.172 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:29:45.887 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:31:46.034 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:31:46.085 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:31:46.814 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:31:46.815 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:31:46.815 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:31:46.815 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:31:46.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:31:47.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:31:50.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:31:50.204 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:31:50.953 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:33:51.100 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:33:51.149 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:33:51.840 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:33:51.841 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:33:51.841 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:33:51.841 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:33:51.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:33:52.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:33:55.176 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:33:55.225 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:33:55.904 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:35:56.051 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:35:56.104 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:35:56.840 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:35:56.840 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:35:56.840 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:35:56.840 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:35:56.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:35:57.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:36:00.137 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:36:00.183 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:36:00.829 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:38:00.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:38:00.982 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:38:01.747 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:38:01.747 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:38:01.747 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:38:01.747 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:38:01.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:38:01.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:38:05.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:38:05.084 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:38:05.858 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:40:06.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:40:06.065 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:40:06.832 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:40:06.833 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:40:06.833 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:40:06.833 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:40:06.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:40:07.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:40:10.117 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:40:10.166 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:40:10.942 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:42:11.107 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:42:11.163 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:42:11.973 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:42:11.973 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:42:11.973 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:42:11.973 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:42:11.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:42:12.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:42:15.273 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:42:15.323 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:42:16.049 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:44:16.208 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:44:16.258 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:44:24.177 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:44:24.177 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:44:24.177 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:44:24.177 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:44:24.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:44:24.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:44:27.460 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:44:27.509 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:44:28.266 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:46:28.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:46:28.479 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:46:29.283 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:46:29.283 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:46:29.283 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:46:29.284 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:46:29.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:46:29.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:46:47.845 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:46:47.896 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:46:48.638 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:48:48.776 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:48:48.827 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:48:49.737 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:48:49.738 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:48:49.738 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:48:49.738 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:48:49.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:48:50.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:48:51.877 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T01:48:51.927 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:48:54.093 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:50:54.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:50:54.300 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:50:55.076 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:50:55.076 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:50:55.076 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:50:55.076 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:50:55.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:50:55.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:50:55.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:50:55.569 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:50:55.747 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:50:57.365 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:50:57.365 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:50:57.399 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:50:57.400 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:50:57.401 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:50:58.863 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:50:58.871 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T01:50:58.923 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:50:59.651 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:52:59.806 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:52:59.866 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:53:00.632 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:53:00.632 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:53:00.632 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:53:00.632 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:53:00.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:53:00.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:53:01.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:53:01.068 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:53:01.243 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:53:01.340 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:53:01.340 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:53:01.366 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:53:01.367 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:53:01.368 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:53:01.435 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:53:01.437 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T01:53:01.491 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:53:02.221 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:55:02.372 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:55:02.424 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:55:03.202 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:55:03.202 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:55:03.202 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:55:03.202 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:55:03.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:55:03.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:55:03.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:55:03.617 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:55:03.943 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T01:55:03.943 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T01:55:03.976 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:55:04.043 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T01:55:04.047 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T01:55:04.096 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:55:04.806 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:57:04.954 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:57:05.020 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:57:05.773 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:57:05.774 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:57:05.774 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:57:05.774 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:57:05.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:57:05.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:57:06.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:57:06.207 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T01:57:24.516 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T01:57:24.566 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:57:25.342 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T01:59:25.478 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T01:59:25.527 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:59:26.270 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T01:59:26.270 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T01:59:26.270 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T01:59:26.271 INFO:teuthology.misc:Compressing logs...
2023-01-16T01:59:26.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T01:59:26.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T01:59:26.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T01:59:26.728 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:27.839 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:28.915 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:30.026 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:31.096 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:32.207 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:33.317 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:34.388 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:35.460 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:36.575 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T01:59:36.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T01:59:36.662 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T01:59:37.403 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:01:37.461 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:01:37.510 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:01:38.229 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:01:38.229 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:01:38.230 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:01:38.230 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:01:38.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:02:38.232 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T02:02:38.285 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:02:38.979 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:04:39.031 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:04:39.085 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:04:39.882 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:04:39.882 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:04:39.882 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:04:39.882 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:04:39.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:04:58.405 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T02:04:58.453 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:04:59.195 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:06:59.248 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:06:59.298 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:07:00.034 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:07:00.034 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:07:00.034 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:07:00.034 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:07:00.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:07:32.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:07:32.232 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:07:32.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:07:32.573 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:07:32.619 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:07:32.620 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:07:32.710 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:07:32.710 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:07:32.712 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:07:32.747 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T02:07:32.749 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:07:32.749 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:07:32.801 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:07:33.515 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:09:33.578 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:09:33.634 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:09:34.422 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:09:34.423 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:09:34.423 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:09:34.423 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:09:34.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:09:34.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:09:34.629 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:09:34.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:09:34.991 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:09:35.022 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:09:35.022 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:09:35.091 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:09:35.092 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:09:35.093 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:09:35.121 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T02:09:35.123 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:09:35.136 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:09:35.190 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:09:35.959 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:11:36.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:11:36.072 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:11:36.818 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:11:36.818 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:11:36.818 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:11:36.819 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:11:36.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:11:37.509 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T02:11:37.559 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:11:38.278 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:13:38.338 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:13:38.387 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:13:39.078 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:13:39.079 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:13:39.079 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:13:39.079 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:13:39.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:13:39.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:13:39.346 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:13:39.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:13:39.770 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:13:39.821 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:13:39.821 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:13:39.891 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:13:39.891 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:13:39.893 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:13:39.923 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:13:39.942 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:13:39.990 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:13:40.743 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:15:40.796 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:15:40.865 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:15:41.622 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:15:41.623 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:15:41.623 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:15:41.623 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:15:41.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:15:41.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:15:41.877 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:15:42.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:15:42.261 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:15:42.306 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:15:42.307 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:15:42.378 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:15:42.378 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:15:42.380 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:15:42.412 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:15:42.427 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:15:42.492 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:15:43.246 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:17:43.300 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:17:43.350 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:17:44.100 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:17:44.100 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:17:44.100 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:17:44.100 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:17:44.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:17:44.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:17:44.356 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:17:44.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:17:44.797 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:17:44.798 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:17:44.874 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:17:44.955 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:17:44.969 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:17:45.025 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:17:45.798 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:19:45.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:19:45.945 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:19:46.687 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:19:46.803 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:19:46.803 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:19:46.803 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:19:46.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:19:46.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:19:47.017 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:19:47.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:19:47.499 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:19:47.605 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:19:47.860 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:19:47.955 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:19:47.971 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:19:47.973 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:19:48.024 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:19:48.779 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:21:48.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:21:48.990 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:21:49.970 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:21:50.027 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:21:50.027 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:21:50.027 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:21:50.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:21:50.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:21:50.354 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:21:50.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:21:51.438 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:21:51.538 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:21:52.509 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:21:52.810 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:21:52.812 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:21:52.863 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:21:53.752 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:23:53.848 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:23:53.896 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:23:54.624 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:23:54.624 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:23:54.624 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:23:54.624 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:23:54.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:24:12.964 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T02:24:13.032 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:24:13.727 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:26:13.779 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:26:13.832 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:26:14.581 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:26:14.581 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:26:14.581 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:26:14.581 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:26:14.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:26:14.698 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:15.810 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:16.886 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:17.999 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:19.071 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:20.183 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:21.259 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:22.372 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:23.483 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:24.598 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:26:24.598 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T02:26:24.646 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:26:25.377 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:28:25.440 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:28:25.497 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:28:26.238 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:28:26.239 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:28:26.239 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:28:26.239 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:28:26.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:28:26.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:28:27.045 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:28:27.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:28:27.204 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:28:27.434 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:28:27.434 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:28:27.461 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:28:27.461 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:28:27.508 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:28:27.535 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T02:28:27.537 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T02:28:27.592 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:28:27.594 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:28:27.641 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:28:28.290 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:30:28.338 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:30:28.386 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:30:29.103 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:30:29.103 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:30:29.103 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:30:29.103 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:30:29.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:30:29.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:30:29.312 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:31:03.845 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T02:31:03.894 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:31:04.573 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:33:04.620 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:33:04.674 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:33:05.399 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:33:05.399 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:33:05.399 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:33:05.399 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:33:05.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:33:05.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:33:05.618 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:33:08.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T02:33:08.696 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:33:09.394 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:35:09.448 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:35:09.502 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:35:10.240 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:35:10.240 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:35:10.241 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:35:10.241 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:35:10.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:35:10.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:35:10.449 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:35:10.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:35:10.621 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:35:10.883 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:35:10.883 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:35:10.918 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:35:10.918 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:35:10.967 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:35:10.995 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T02:35:11.016 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:35:11.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:35:11.073 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:35:11.814 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:37:11.862 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:37:11.911 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:37:12.635 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:37:12.636 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:37:12.636 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:37:12.636 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:37:12.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:37:12.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:37:12.843 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:38:12.819 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T02:38:12.933 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:38:13.706 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:40:13.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:40:13.813 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:40:14.639 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:40:14.639 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:40:14.639 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:40:14.639 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:40:14.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:40:14.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:40:14.849 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:40:15.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:40:15.073 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:40:15.328 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:40:15.328 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:40:15.377 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:40:15.377 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:40:15.429 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:40:15.458 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:40:15.459 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:40:15.512 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:40:16.247 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:42:16.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:42:16.354 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:42:17.098 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:42:17.098 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:42:17.098 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:42:17.098 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:42:17.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:42:17.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:42:17.319 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:42:17.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:42:17.544 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:42:17.782 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:42:17.783 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:42:17.825 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:42:17.825 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:42:17.873 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:42:17.905 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:42:17.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:42:17.960 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:42:18.654 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:44:18.710 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:44:18.760 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:44:19.502 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:44:19.502 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:44:19.502 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:44:19.502 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:44:19.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:44:19.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:44:19.712 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:44:19.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:44:19.933 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:44:20.182 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:44:20.182 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:44:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:44:20.225 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:44:20.272 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:44:20.302 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:44:20.315 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:44:20.363 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:44:21.059 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:46:21.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:46:21.169 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:46:21.909 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:46:21.909 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:46:21.909 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:46:21.910 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:46:21.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:46:22.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:46:22.083 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:46:22.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:46:22.305 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:46:22.514 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:46:22.515 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:46:22.557 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:46:22.557 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:46:22.605 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:46:22.634 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:46:22.648 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:46:22.715 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:46:23.459 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:48:23.516 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:48:23.567 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:48:24.357 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:48:24.358 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:48:24.358 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:48:24.358 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:48:24.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:48:24.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:48:24.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:48:24.814 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:48:25.041 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:48:25.041 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:48:25.087 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:48:25.115 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:48:25.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:48:25.180 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:48:25.930 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:50:25.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:50:26.046 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:50:26.794 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:50:26.794 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:50:26.794 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:50:26.794 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:50:26.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:50:26.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:50:44.841 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T02:50:44.890 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:50:45.636 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:52:45.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:52:45.735 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:52:46.499 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:52:46.500 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:52:46.500 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:52:46.500 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:52:46.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:52:46.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:52:46.753 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:47.828 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:48.941 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:50.051 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:51.121 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:52.230 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:53.304 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:55.796 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:56.869 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:57.981 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T02:52:57.981 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T02:52:58.029 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:52:58.782 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:54:58.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:54:58.891 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:54:59.645 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:54:59.646 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:54:59.646 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:54:59.646 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:54:59.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:54:59.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:55:02.916 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T02:55:02.967 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:55:03.661 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:57:03.709 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:57:03.810 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:57:04.579 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:57:04.579 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:57:04.579 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:57:04.579 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:57:04.579 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:57:04.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:57:04.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T02:57:04.952 INFO:teuthology.orchestra.run.smithi142.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T02:57:37.017 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T02:57:37.087 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:57:37.087 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:57:37.128 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T02:57:37.128 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T02:57:37.130 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:57:37.148 INFO:teuthology.orchestra.run.smithi142.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T02:57:37.163 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T02:57:37.171 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T02:57:37.230 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:57:37.946 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T02:59:37.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T02:59:38.047 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:59:38.814 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T02:59:38.814 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T02:59:38.815 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T02:59:38.815 INFO:teuthology.misc:Compressing logs...
2023-01-16T02:59:38.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T02:59:38.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T02:59:57.482 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T02:59:57.560 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T02:59:58.307 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:01:58.362 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:01:58.421 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:01:59.138 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:01:59.139 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:01:59.139 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:01:59.139 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:01:59.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:02:17.508 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:02:17.558 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:02:18.303 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:04:18.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:04:18.408 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:04:19.186 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:04:19.186 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:04:19.186 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:04:19.186 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:04:19.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:04:22.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:04:22.303 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:04:23.020 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:06:23.084 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:06:23.139 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:06:23.871 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:06:23.872 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:06:23.872 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:06:23.872 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:06:23.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:06:26.952 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:06:27.003 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:06:27.704 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:08:27.757 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:08:27.809 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:08:28.536 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:08:28.536 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:08:28.536 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:08:28.537 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:08:28.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:08:31.593 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:08:31.642 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:08:32.329 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:10:32.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:10:32.437 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:10:33.159 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:10:33.160 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:10:33.160 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:10:33.160 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:10:33.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:10:40.409 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:41.487 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:42.603 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:43.721 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:44.839 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:45.955 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:47.072 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:48.187 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:49.303 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:50.379 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:10:50.380 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:10:50.429 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:10:51.142 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:12:51.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:12:51.243 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:12:51.969 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:12:51.969 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:12:51.969 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:12:51.969 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:12:51.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:13:07.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:13:07.471 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:13:07.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:13:08.057 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:13:08.058 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:13:08.109 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:13:08.136 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T03:13:08.176 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T03:13:08.208 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T03:13:08.208 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:13:08.286 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T03:13:08.286 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:13:08.287 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:13:08.314 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T03:13:08.419 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping
2023-01-16T03:13:15.679 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping
2023-01-16T03:13:21.642 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping
2023-01-16T03:13:28.663 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping
2023-01-16T03:13:35.764 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping
2023-01-16T03:13:35.800 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T03:13:35.800 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:13:43.254 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:13:43.981 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:15:44.043 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:15:44.097 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:15:44.892 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:15:44.892 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:15:44.892 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:15:44.892 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:15:44.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:15:45.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:15:45.099 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:15:45.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:15:45.722 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:15:45.723 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:15:45.780 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:15:45.853 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T03:15:45.997 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T03:15:45.997 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:15:45.998 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:15:46.025 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T03:15:46.095 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T03:15:46.095 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:15:53.087 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:15:53.883 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:17:53.936 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:17:53.987 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:17:54.727 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:17:54.727 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:17:54.727 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:17:54.727 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:17:54.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:17:54.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:17:54.893 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:17:55.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:18:13.669 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T03:18:13.721 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:18:14.439 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:20:14.491 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:20:14.539 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:20:15.326 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:20:15.327 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:20:15.327 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:20:15.327 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:20:15.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:20:15.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:20:15.490 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:20:15.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:20:34.472 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T03:20:34.523 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:20:35.217 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:22:35.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:22:35.340 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:22:36.082 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:22:36.083 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:22:36.083 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:22:36.083 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:22:36.083 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:22:36.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:22:36.347 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:37.425 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:38.544 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:39.623 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:40.701 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:41.779 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:42.896 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:43.974 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:45.091 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:46.171 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:22:46.171 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T03:22:46.221 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:22:46.938 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:24:46.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:24:47.048 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:24:47.752 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:24:47.752 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:24:47.753 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:24:47.753 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:24:47.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:24:47.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:24:55.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:24:55.852 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:25:14.276 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T03:25:14.326 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:25:15.006 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:27:15.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:27:15.107 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:27:15.869 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:27:15.869 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:27:15.869 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:27:15.870 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:27:15.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:27:16.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:27:50.693 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T03:27:50.740 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:27:51.454 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:29:51.508 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:29:51.560 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:29:52.350 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:29:52.351 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:29:52.351 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:29:52.351 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:29:52.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:30:10.724 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:30:10.778 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:30:11.520 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:32:11.577 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:32:11.626 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:32:12.361 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:32:12.361 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:32:12.361 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:32:12.361 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:32:12.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:32:15.429 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:32:15.478 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:32:16.174 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:34:16.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:34:16.278 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:34:17.019 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:34:17.019 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:34:17.019 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:34:17.019 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:34:17.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:34:20.100 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:34:20.146 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:34:20.897 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:36:20.956 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:36:21.049 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:36:21.779 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:36:21.779 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:36:21.779 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:36:21.779 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:36:21.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:36:24.836 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:36:24.886 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:36:25.568 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:38:25.621 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:38:25.676 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:38:26.421 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:38:26.421 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:38:26.422 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:38:26.422 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:38:26.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:38:26.539 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:27.654 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:28.767 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:29.881 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:30.994 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:32.108 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:33.221 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:34.335 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:35.452 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:36.529 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:38:36.529 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:38:36.586 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:38:37.345 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:40:37.399 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:40:37.449 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:40:38.166 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:40:38.166 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:40:38.166 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:40:38.167 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:40:38.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:40:41.221 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T03:40:41.266 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:40:41.963 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:42:42.026 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:42:42.075 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:42:42.840 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:42:42.841 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:42:42.841 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:42:42.841 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:42:42.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:42:43.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:42:43.036 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T03:42:43.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:42:43.421 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:42:44.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T03:42:44.183 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:42:44.901 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:44:44.958 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:44:45.004 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:44:45.730 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:44:45.730 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:44:45.731 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:44:45.731 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:44:45.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:44:45.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:44:45.918 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T03:44:46.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:44:46.335 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:44:46.368 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:44:46.368 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:44:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:44:46.526 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:44:46.527 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:44:46.544 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T03:44:46.593 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T03:44:46.668 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T03:44:46.668 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:44:46.816 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T03:44:46.817 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:44:46.818 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:44:46.835 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T03:44:46.883 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T03:44:46.989 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T03:44:46.990 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:44:47.200 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T03:44:47.200 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:44:47.250 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:44:47.934 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:46:47.994 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:46:48.050 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:46:48.786 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:46:48.786 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:46:48.786 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:46:48.786 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:46:48.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:46:48.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:46:48.967 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T03:46:49.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:46:59.365 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T03:46:59.416 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:47:00.207 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:49:00.260 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:49:00.318 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:49:01.058 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:49:01.058 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:49:01.058 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:49:01.058 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:49:01.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:49:01.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:49:01.241 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T03:49:01.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:49:01.719 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:49:01.740 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:49:01.740 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:49:01.894 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:49:01.895 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:49:01.896 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:49:01.992 INFO:teuthology.orchestra.run.smithi142.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T03:49:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T03:49:02.130 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:49:02.199 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:49:02.215 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T03:49:02.283 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T03:49:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.5.log.gz already exists;	not overwritten
2023-01-16T03:49:02.305 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.6.log.gz already exists;	not overwritten
2023-01-16T03:49:02.306 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.4.log.gz already exists;	not overwritten
2023-01-16T03:49:02.306 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-osd.7.log.gz already exists;	not overwritten
2023-01-16T03:49:02.311 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T03:49:02.366 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:49:03.065 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:51:03.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:51:03.171 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:51:03.884 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:51:03.885 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:51:03.885 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:51:03.885 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:51:03.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:51:04.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:51:04.073 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T03:51:19.537 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:20.655 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:21.773 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:22.851 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:23.929 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:25.048 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:26.131 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:27.249 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:28.327 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:29.446 ERROR:teuthology.orchestra.connection:Error authenticating with smithi142.front.sepia.ceph.com: Authentication failed.
2023-01-16T03:51:29.446 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T03:51:29.494 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:51:30.171 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:53:30.235 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:53:30.289 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:53:31.079 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:53:31.079 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:53:31.079 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:53:31.080 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:53:31.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:53:31.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:53:31.273 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T03:53:34.340 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T03:53:34.391 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:53:35.115 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:55:35.177 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:55:35.229 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:55:35.970 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:55:35.970 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:55:35.970 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:55:35.970 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:55:35.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:55:36.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:55:36.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T03:55:36.354 INFO:teuthology.orchestra.run.smithi142.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T03:55:36.631 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T03:55:36.682 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:55:36.683 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:55:36.726 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T03:55:36.727 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T03:55:36.728 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:55:36.746 INFO:teuthology.orchestra.run.smithi142.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T03:55:36.813 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T03:55:36.815 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T03:55:36.865 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:55:37.640 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T03:57:37.692 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T03:57:37.746 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:57:38.582 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T03:57:38.582 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T03:57:38.582 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T03:57:38.582 INFO:teuthology.misc:Compressing logs...
2023-01-16T03:57:38.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T03:57:38.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T03:58:13.413 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi142
2023-01-16T03:58:13.464 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T03:58:14.177 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:00:14.230 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:00:14.283 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:00:15.038 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:00:15.038 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:00:15.038 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:00:15.038 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:00:15.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:00:15.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:00:15.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:00:15.633 INFO:teuthology.orchestra.run.smithi142.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:00:15.894 INFO:teuthology.orchestra.run.smithi142.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:00:15.895 INFO:teuthology.orchestra.run.smithi142.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:00:15.944 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:00:15.995 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T04:00:15.997 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:00:16.047 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:00:16.751 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:02:16.804 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:02:16.859 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:02:17.598 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:02:17.598 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:02:17.598 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:02:17.598 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:02:17.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:02:17.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:02:18.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:02:18.084 INFO:teuthology.orchestra.run.smithi142.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:02:36.581 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T04:02:36.639 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:02:37.388 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:04:37.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:04:37.554 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:04:38.323 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:04:38.324 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:04:38.324 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:04:38.324 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:04:38.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:04:38.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:04:38.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:04:38.840 INFO:teuthology.orchestra.run.smithi142.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:04:38.896 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:04:39.973 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:04:41.089 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:04:42.204 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:04:43.320 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:04:44.434 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:04:45.549 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:04:46.838 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:05:47.840 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T04:05:47.893 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:05:48.569 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:07:48.631 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:07:48.684 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:07:49.412 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:07:49.412 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:07:49.412 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:07:49.412 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:07:49.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:07:49.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:07:49.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:07:50.152 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:07:50.230 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:07:50.230 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:07:50.232 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:07:50.424 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:07:50.474 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T04:07:50.860 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping
2023-01-16T04:07:51.365 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping
2023-01-16T04:07:51.582 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping
2023-01-16T04:07:52.234 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T04:07:52.234 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:07:52.284 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:07:53.062 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:09:53.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:09:53.170 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:09:53.959 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:09:53.959 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:09:53.959 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:09:53.960 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:09:53.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:09:54.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:09:54.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:09:54.663 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:09:54.735 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:09:54.735 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:09:54.737 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:09:54.917 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:09:54.972 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T04:09:55.011 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping
2023-01-16T04:09:56.566 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-16T04:09:56.567 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:09:56.619 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:09:57.357 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:11:57.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:11:57.468 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:11:58.221 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:11:58.221 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:11:58.221 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:11:58.221 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:11:58.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:11:58.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:11:58.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:11:58.924 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:11:58.996 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:11:58.996 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:11:58.997 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:11:59.179 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:11:59.212 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T04:11:59.215 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:11:59.320 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:11:59.992 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:14:00.047 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:14:00.098 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:14:00.869 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:14:00.869 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:14:00.870 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:14:00.870 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:14:00.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:14:01.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:14:01.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:14:01.953 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:14:02.029 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:14:02.030 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:14:02.031 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:14:02.100 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T04:14:02.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:14:02.156 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:14:02.882 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:16:02.944 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:16:02.996 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:16:03.710 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:16:03.710 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:16:03.710 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:16:03.710 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:16:03.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:16:03.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:16:04.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:16:04.526 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:16:04.593 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T04:16:04.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:16:04.646 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:16:05.358 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:18:05.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:18:05.500 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:18:06.244 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:18:06.244 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:18:06.244 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:18:06.245 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:18:06.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:18:06.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:18:06.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:18:07.016 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:18:07.086 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T04:18:07.087 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:18:07.137 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:18:07.889 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:20:07.945 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:20:07.993 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:20:08.765 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:20:08.765 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:20:08.765 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:20:08.765 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:20:08.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:20:09.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:20:09.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:20:27.941 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T04:20:27.995 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:20:28.698 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:22:28.755 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:22:28.807 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:22:29.561 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:22:29.562 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:22:29.562 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:22:29.562 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:22:29.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:22:29.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:22:30.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:22:33.253 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T04:22:33.300 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:22:33.984 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:24:34.039 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:24:34.087 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:24:34.837 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:24:34.837 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:24:34.837 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:24:34.837 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:24:34.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:24:53.157 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T04:24:53.223 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:24:53.951 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:26:54.001 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:26:54.054 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:26:54.788 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:26:54.788 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:26:54.788 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:26:54.789 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:26:54.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:26:54.871 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:26:55.990 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:26:57.071 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:26:58.188 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:26:59.307 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:27:00.425 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:27:01.542 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:27:02.661 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:27:03.780 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:27:04.898 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:27:04.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T04:27:04.951 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:27:05.688 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:29:05.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:29:05.813 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:29:06.585 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:29:06.586 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:29:06.586 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:29:06.586 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:29:06.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:29:07.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:29:07.385 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:29:07.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:29:07.753 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:29:07.800 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:29:07.800 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:29:07.848 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:29:07.849 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:29:07.851 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:29:07.883 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T04:29:07.892 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T04:29:07.955 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T04:29:07.956 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:29:08.013 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T04:29:08.013 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:29:08.015 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:29:08.046 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T04:29:08.056 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T04:29:08.064 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T04:29:08.067 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:29:08.117 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:29:08.871 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:31:08.926 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:31:09.025 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:31:09.766 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:31:09.766 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:31:09.767 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:31:09.767 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:31:09.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:31:09.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:31:09.975 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:31:10.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:31:10.270 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T04:31:10.321 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:31:11.033 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:33:11.082 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:33:11.135 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:33:11.899 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:33:11.899 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:33:11.900 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:33:11.900 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:33:11.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:33:14.985 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T04:33:15.044 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:33:15.775 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:35:15.829 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:35:15.879 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:35:16.637 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:35:16.637 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:35:16.637 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:35:16.637 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:35:16.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:35:16.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:35:16.900 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:35:17.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:35:17.510 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:35:17.567 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:35:17.568 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:35:17.616 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:35:17.617 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:35:17.620 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:35:17.852 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:35:17.880 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T04:35:17.894 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T04:35:17.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:35:17.949 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:35:18.676 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:37:18.725 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:37:18.779 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:37:19.505 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:37:19.506 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:37:19.506 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:37:19.506 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:37:19.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:37:19.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:37:19.770 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:37:20.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:37:20.370 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:37:20.415 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:37:20.415 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:37:20.465 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:37:20.466 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:37:20.468 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:37:20.673 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:37:20.702 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T04:37:20.715 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T04:37:20.720 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:37:20.780 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:37:21.459 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:39:21.517 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:39:21.570 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:39:22.295 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:39:22.295 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:39:22.295 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:39:22.295 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:39:22.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:39:22.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:39:22.546 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:39:22.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:39:23.212 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:39:23.212 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:39:23.265 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:39:23.473 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:39:23.502 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T04:39:23.517 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T04:39:23.521 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:39:23.570 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:39:24.254 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:41:24.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:41:24.355 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:41:25.106 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:41:25.106 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:41:25.106 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:41:25.106 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:41:25.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:41:25.285 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:41:25.328 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:41:25.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:41:25.987 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:41:25.987 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:41:26.046 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:41:26.093 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T04:41:26.095 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:41:26.143 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:41:26.828 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:43:26.882 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:43:26.936 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:43:27.663 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:43:27.664 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:43:27.664 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:43:27.664 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:43:27.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:43:27.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:43:27.941 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T04:43:28.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:43:28.566 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:43:28.566 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:43:28.611 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:43:28.660 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T04:43:28.662 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:43:28.713 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:43:29.411 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:45:29.466 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:45:29.516 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:45:30.242 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:45:30.243 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:45:30.243 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:45:30.243 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:45:30.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:45:48.581 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T04:45:48.634 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:45:49.322 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:47:49.374 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:47:49.426 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:47:50.185 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:47:50.186 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:47:50.186 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:47:50.186 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:47:50.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:47:57.465 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:47:58.580 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:47:59.696 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:00.811 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:01.926 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:03.043 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:04.159 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:05.272 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:06.389 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:07.466 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:48:07.466 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T04:48:07.514 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:48:08.214 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:50:08.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:50:08.321 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:50:09.045 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:50:09.045 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:50:09.045 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:50:09.045 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:50:09.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:50:09.317 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T04:50:09.369 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:50:10.077 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:52:10.130 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:52:10.183 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:52:10.928 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:52:10.928 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:52:10.928 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:52:10.928 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:52:10.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:52:11.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:52:11.217 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:52:11.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:52:11.882 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:52:11.882 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:52:11.940 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:52:11.956 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T04:52:12.026 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T04:52:12.026 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:52:12.103 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:52:12.119 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T04:52:12.130 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T04:52:12.152 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:52:12.207 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:52:12.920 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:54:12.974 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:54:13.027 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:54:13.829 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:54:13.830 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:54:13.830 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:54:13.830 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:54:13.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:54:14.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:54:14.036 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:54:14.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:54:14.847 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T04:54:14.847 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:54:14.905 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:54:14.922 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T04:54:14.971 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T04:54:14.992 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T04:54:14.992 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:54:15.055 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T04:54:15.055 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T04:54:15.059 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:54:15.074 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T04:54:15.102 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T04:54:15.106 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T04:54:15.158 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:54:15.939 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:56:16.012 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:56:16.059 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:56:16.828 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:56:16.829 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:56:16.829 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:56:16.829 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:56:16.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:56:17.783 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T04:56:17.839 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:56:18.595 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T04:58:18.656 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T04:58:18.709 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:58:19.446 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T04:58:19.447 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T04:58:19.447 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T04:58:19.447 INFO:teuthology.misc:Compressing logs...
2023-01-16T04:58:19.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T04:58:19.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T04:58:19.785 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T04:58:20.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T04:58:20.157 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:21.273 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:22.390 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:23.505 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:24.620 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:25.734 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:26.849 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:27.964 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:29.079 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:30.196 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T04:58:30.197 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T04:58:30.246 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T04:58:30.967 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:00:31.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:00:31.071 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:00:31.813 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:00:31.814 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:00:31.814 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:00:31.814 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:00:31.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:00:32.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:00:32.147 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T05:00:32.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:00:32.662 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:00:32.693 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:00:32.693 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:00:32.758 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:00:32.758 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:00:32.759 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:00:32.825 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T05:00:32.939 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T05:00:32.940 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:00:32.941 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:00:32.962 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T05:00:32.979 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T05:00:32.982 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:00:33.035 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:00:33.776 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:02:33.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:02:33.893 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:02:34.681 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:02:34.681 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:02:34.681 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:02:34.681 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:02:34.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:02:34.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:02:34.974 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T05:02:35.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:02:35.486 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:02:35.509 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:02:35.509 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:02:35.564 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:02:35.564 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:02:35.565 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:02:35.748 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:02:35.770 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T05:02:35.779 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T05:02:35.782 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:02:35.833 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:02:36.535 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:04:36.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:04:36.646 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:04:37.376 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:04:37.376 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:04:37.376 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:04:37.376 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:04:37.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:04:37.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:04:38.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:04:38.221 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:04:38.303 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:04:38.303 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:04:38.304 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:04:38.485 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:04:38.516 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T05:04:38.526 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:04:38.609 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:04:39.303 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:06:39.386 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:06:39.438 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:06:40.180 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:06:40.181 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:06:40.181 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:06:40.181 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:06:40.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:06:40.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:06:40.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:06:41.039 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:06:41.121 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:06:41.121 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:06:41.122 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:06:41.190 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:06:41.192 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:06:41.242 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:06:42.009 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:08:42.060 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:08:42.116 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:08:42.884 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:08:42.884 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:08:42.885 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:08:42.885 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:08:42.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:08:43.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:08:43.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:08:43.736 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:08:43.817 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:08:43.818 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:08:43.819 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:08:43.886 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:08:43.887 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:08:43.936 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:08:44.599 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:10:44.658 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:10:44.716 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:10:45.479 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:10:45.479 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:10:45.479 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:10:45.480 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:10:45.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:10:45.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:10:46.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:10:46.402 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:10:46.471 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:10:46.473 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:10:46.523 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:10:47.234 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:12:47.286 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:12:47.339 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:12:48.104 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:12:48.105 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:12:48.105 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:12:48.105 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:12:48.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:12:48.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:12:48.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:12:49.194 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:12:49.221 INFO:teuthology.orchestra.run.smithi019.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T05:12:49.264 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:12:49.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:12:49.324 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:12:50.087 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:14:50.145 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:14:50.196 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:14:51.007 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:14:51.007 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:14:51.007 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:14:51.007 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:14:51.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:14:51.095 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:52.208 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:53.279 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:54.391 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:55.504 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:56.576 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:57.687 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:58.800 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:14:59.874 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:15:00.986 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:15:00.986 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T05:15:01.037 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:15:07.153 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:17:07.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:17:07.276 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:17:11.856 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:17:11.856 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:17:11.856 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:17:11.856 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:17:11.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:17:12.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:17:12.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:17:12.408 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:17:12.583 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:17:12.583 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:17:12.644 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:17:12.714 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:17:12.715 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-16T05:17:12.762 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T05:17:12.774 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:17:12.825 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:17:13.547 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:19:14.396 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:19:14.448 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:19:15.177 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:19:15.178 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:19:15.178 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:19:15.178 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:19:15.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:19:15.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:19:15.340 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:19:15.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:19:15.866 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:19:15.866 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:19:15.927 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:19:15.954 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T05:19:15.998 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:19:15.998 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-16T05:19:15.999 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:19:16.047 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:19:16.813 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:21:16.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:21:16.921 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:21:19.241 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:21:19.242 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:21:19.242 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:21:19.242 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:21:19.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:21:53.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T05:21:53.876 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:21:54.626 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:23:54.688 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:23:54.788 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:23:55.593 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:23:55.594 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:23:55.594 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:23:55.594 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:23:55.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:23:55.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:23:55.846 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:23:56.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:24:14.628 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T05:24:14.680 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:24:15.393 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:26:15.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:26:15.499 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:26:16.273 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:26:16.273 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:26:16.273 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:26:16.273 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:26:16.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:26:16.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:26:16.526 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:26:16.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:26:32.217 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:33.331 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:34.443 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:35.557 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:36.670 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:37.783 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:38.896 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:39.973 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:41.086 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:42.162 ERROR:teuthology.orchestra.connection:Error authenticating with smithi146.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:26:42.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T05:26:42.213 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:26:42.989 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:28:43.043 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:28:43.094 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:28:43.807 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:28:43.808 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:28:43.808 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:28:43.808 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:28:43.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:28:44.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:28:44.071 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:28:44.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:28:47.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T05:28:47.507 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:28:48.202 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:30:48.252 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:30:48.303 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:30:49.050 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:30:49.051 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:30:49.051 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:30:49.051 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:30:49.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:30:49.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:30:49.306 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:30:49.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:30:49.774 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T05:30:49.833 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:30:49.833 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:30:49.888 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:30:49.889 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:30:49.891 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:30:49.922 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T05:30:50.062 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T05:30:50.062 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:30:50.066 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:30:50.110 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T05:30:50.112 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T05:30:50.114 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:30:50.169 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:30:50.853 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:32:50.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:32:50.967 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:32:51.737 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:32:51.737 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:32:51.737 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:32:51.737 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:32:51.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:32:51.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:32:52.006 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:32:52.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:32:52.452 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T05:32:52.504 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:32:52.504 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:32:52.551 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:32:52.551 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:32:52.553 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:32:52.608 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-16T05:32:52.725 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T05:32:52.725 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:32:52.727 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:32:52.770 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T05:32:52.773 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/log/ceph': No such file or directory
2023-01-16T05:32:52.785 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T05:32:52.789 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:32:52.841 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:32:53.550 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:34:53.611 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:34:53.664 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:34:54.428 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:34:54.429 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:34:54.429 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:34:54.429 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:34:54.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:35:12.808 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T05:35:12.863 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:35:13.575 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:37:13.630 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:37:13.690 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:37:14.452 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:37:14.452 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:37:14.452 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:37:14.452 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:37:14.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:37:14.535 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:15.652 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:16.770 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:17.895 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:19.015 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:20.134 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:21.255 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:22.373 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:23.489 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:24.775 ERROR:teuthology.orchestra.connection:Error authenticating with smithi019.front.sepia.ceph.com: Authentication failed.
2023-01-16T05:37:24.776 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T05:37:24.825 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:37:25.544 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:39:25.599 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:39:25.647 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:39:26.381 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:39:26.382 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:39:26.382 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:39:26.382 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:39:26.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:39:26.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:39:26.691 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:39:26.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:40:26.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi146
2023-01-16T05:40:27.000 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:40:27.742 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:42:27.799 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:42:27.857 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:42:28.691 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:42:28.691 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:42:28.691 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:42:28.691 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:42:28.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:42:28.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:42:28.891 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:42:29.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:42:29.614 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T05:42:29.665 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:42:29.665 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:42:29.736 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:42:29.737 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:42:29.741 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:42:29.816 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-01-16T05:42:29.940 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:42:29.970 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T05:42:29.971 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T05:42:29.983 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:42:30.038 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:42:30.783 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:44:30.836 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:44:30.892 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:44:31.635 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:44:31.635 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:44:31.635 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:44:31.635 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:44:31.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:44:31.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:44:31.840 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:44:32.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:44:32.461 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T05:44:32.493 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:44:32.493 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:44:32.545 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:44:32.546 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:44:32.550 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:44:32.738 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:44:32.765 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T05:44:32.769 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T05:44:32.784 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:44:32.836 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:44:33.555 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:46:33.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:46:33.662 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:46:34.387 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:46:34.387 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:46:34.387 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:46:34.388 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:46:34.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:46:34.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:46:34.552 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:46:34.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:46:35.137 INFO:teuthology.orchestra.run.smithi146.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-16T05:46:35.167 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:46:35.167 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:46:35.218 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:46:35.218 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:46:35.220 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:46:35.248 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:46:35.265 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:46:35.328 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:46:36.043 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:48:36.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:48:36.154 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:48:36.873 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:48:36.873 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:48:36.873 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:48:36.874 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:48:36.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:48:37.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:48:37.083 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T05:48:37.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:48:37.703 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T05:48:37.704 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T05:48:37.755 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:48:37.783 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:48:37.798 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:48:37.848 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:48:38.523 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:50:38.578 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:50:38.629 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:50:39.394 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:50:39.394 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:50:39.394 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:50:39.395 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:50:39.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:50:39.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:50:39.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:50:40.298 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:50:40.328 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:50:40.347 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:50:40.400 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:50:41.113 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:52:41.176 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:52:41.225 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:52:41.942 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:52:41.942 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:52:41.943 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:52:41.943 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:52:41.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:52:42.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:52:42.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:52:42.849 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:52:42.878 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:52:42.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:52:42.946 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:52:43.594 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:54:43.650 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:54:43.703 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:54:44.398 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:54:44.398 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:54:44.398 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:54:44.398 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:54:44.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:54:44.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T05:54:44.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T05:54:45.326 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:54:45.360 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T05:54:45.370 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T05:54:45.420 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:54:46.114 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:56:46.192 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:56:46.238 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:56:47.013 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:56:47.014 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:56:47.014 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:56:47.014 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:56:47.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T05:57:05.573 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T05:57:05.623 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:57:06.330 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T05:59:06.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T05:59:06.449 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T05:59:07.189 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T05:59:07.190 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T05:59:07.190 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T05:59:07.190 INFO:teuthology.misc:Compressing logs...
2023-01-16T05:59:07.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:00:07.193 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T06:00:07.301 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:00:08.005 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:02:08.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:02:08.112 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:02:08.872 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:02:08.872 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:02:08.872 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:02:08.872 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:02:08.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:02:09.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:02:09.038 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:02:09.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:02:09.535 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:02:09.583 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:02:09.583 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:02:09.651 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:02:09.651 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:02:09.652 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:02:09.679 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T06:02:09.719 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-16T06:02:09.747 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T06:02:09.747 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:02:09.811 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-16T06:02:09.812 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:02:09.813 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:02:09.839 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:02:09.852 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T06:02:09.855 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:02:09.906 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:02:10.614 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:04:10.673 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:04:10.726 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:04:11.550 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:04:11.550 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:04:11.550 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:04:11.550 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:04:11.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:04:17.669 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi019
2023-01-16T06:04:17.730 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:04:18.448 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:06:18.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:06:18.560 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:06:19.319 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:06:19.319 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:06:19.319 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:06:19.320 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:06:19.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:06:19.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:06:19.651 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:06:19.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:06:20.083 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:06:20.194 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:06:20.194 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:06:20.257 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:06:20.257 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:06:20.258 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:06:20.559 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:06:20.593 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T06:06:20.626 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:06:20.657 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:06:20.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:06:20.714 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:06:21.444 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:08:21.507 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:08:21.559 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:08:22.333 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:08:22.334 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:08:22.334 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:08:22.334 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:08:22.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:08:22.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:08:22.671 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:08:22.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:08:23.107 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:08:23.206 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:08:23.206 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:08:23.268 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:08:23.268 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:08:23.270 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:08:23.572 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:08:23.606 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T06:08:23.638 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:08:23.668 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:08:23.679 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:08:23.724 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:08:24.466 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:10:24.518 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:10:24.569 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:10:25.290 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:10:25.290 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:10:25.290 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:10:25.290 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:10:25.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:10:25.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:10:25.618 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:10:25.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:10:26.064 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:10:26.163 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:10:26.163 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:10:26.223 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:10:26.223 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:10:26.224 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:10:26.526 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:10:26.562 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T06:10:26.592 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:10:26.624 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:10:26.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:10:26.683 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:10:27.349 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:12:27.402 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:12:27.458 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:12:28.167 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:12:28.167 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:12:28.167 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:12:28.168 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:12:28.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:12:28.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:12:28.500 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:12:28.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:12:28.887 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:12:28.983 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:12:28.983 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:12:29.043 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:12:29.044 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:12:29.045 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:12:29.327 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:12:29.365 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists;	not overwritten
2023-01-16T06:12:29.393 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:12:29.422 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-16T06:12:29.435 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi142 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:12:29.484 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:12:30.200 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:14:30.265 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:14:30.317 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:14:31.038 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:14:31.038 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:14:31.039 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:14:31.039 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:14:31.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:14:31.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:14:31.387 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:14:31.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:14:31.815 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:14:31.915 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:14:31.915 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:14:31.976 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:14:31.976 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:14:31.977 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:14:32.084 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:14:32.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:14:32.142 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:14:32.890 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:16:32.960 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:16:33.016 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:16:33.741 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:16:33.742 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:16:33.742 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:16:33.742 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:16:33.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:16:33.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:16:34.076 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:16:34.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:16:34.458 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:16:34.561 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:16:34.562 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:16:34.618 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:16:34.618 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:16:34.619 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:16:34.726 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:16:34.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:16:34.782 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:16:35.459 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:18:35.515 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:18:35.639 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:18:36.349 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:18:36.349 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:18:36.349 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:18:36.349 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:18:36.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:18:36.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:18:36.721 INFO:teuthology.orchestra.run.smithi019.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:18:36.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:18:37.123 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:18:37.222 INFO:teuthology.orchestra.run.smithi019.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:18:37.222 INFO:teuthology.orchestra.run.smithi019.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:18:37.287 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:18:37.287 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:18:37.288 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:18:37.395 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:18:37.402 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:18:37.454 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:18:38.164 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:20:38.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:20:38.270 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:20:39.002 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:20:39.002 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:20:39.002 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:20:39.002 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:20:39.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:20:39.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:20:39.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:20:39.776 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:20:39.943 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:20:39.943 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:20:39.944 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:20:40.054 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:20:40.060 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:20:40.107 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:20:40.789 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:22:40.852 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:22:40.899 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:22:41.641 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:22:41.642 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:22:41.642 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:22:41.642 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:22:41.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:22:41.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:22:42.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:22:42.391 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:22:42.568 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:22:42.568 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:22:42.569 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:22:42.676 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:22:42.682 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:22:42.734 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:22:43.420 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:24:43.490 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:24:43.544 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:24:44.308 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:24:44.309 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:24:44.309 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:24:44.309 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:24:44.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:24:44.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:24:44.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:24:45.045 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:24:45.210 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:24:45.210 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:24:45.211 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:24:45.339 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:24:45.345 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:24:45.396 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:24:46.130 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:26:46.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:26:46.269 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:26:47.038 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:26:47.039 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:26:47.039 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:26:47.039 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:26:47.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:26:47.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:26:47.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:26:47.813 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:26:47.982 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:26:47.982 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:26:47.983 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:26:48.089 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:26:48.095 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:26:48.148 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:26:48.899 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:28:48.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:28:49.031 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:28:49.790 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:28:49.790 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:28:49.790 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:28:49.791 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:28:49.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:28:50.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:28:50.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:28:50.543 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:28:50.706 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:28:50.707 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:28:50.708 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:28:50.812 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:28:50.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:28:50.875 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:28:51.664 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:30:51.730 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:30:51.804 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:30:52.556 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:30:52.556 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:30:52.556 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:30:52.556 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:30:52.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:30:52.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:30:53.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:30:53.325 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:30:53.489 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:30:53.490 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:30:53.491 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:30:53.594 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:30:53.599 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:30:53.651 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:30:54.399 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:32:54.461 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:32:54.514 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:32:55.285 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:32:55.286 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:32:55.286 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:32:55.286 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:32:55.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:32:55.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:32:55.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:32:56.007 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:32:56.176 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:32:56.176 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:32:56.177 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:32:56.289 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:32:56.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:32:56.347 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:32:57.148 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:34:57.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:34:57.261 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:34:58.068 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:34:58.068 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:34:58.068 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:34:58.069 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:34:58.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:34:58.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:34:58.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:34:58.810 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:34:58.970 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:34:58.971 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:34:58.972 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:34:59.073 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:34:59.079 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:34:59.132 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:34:59.835 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:36:59.889 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-16T06:36:59.947 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:37:00.725 INFO:teuthology.task.internal:roles: ubuntu@smithi019.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
2023-01-16T06:37:00.726 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
2023-01-16T06:37:00.726 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['client.0']
2023-01-16T06:37:00.726 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:37:00.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi019.front.sepia.ceph.com'
2023-01-16T06:37:00.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-01-16T06:37:01.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com'
2023-01-16T06:37:01.482 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-16T06:37:01.642 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-16T06:37:01.642 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now
2023-01-16T06:37:01.643 INFO:teuthology.misc:Compressing logs...
2023-01-16T06:37:01.744 INFO:teuthology.orchestra.run.smithi019.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-16T06:37:01.750 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi019 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-16T06:37:01.804 INFO:teuthology.kill:Killing Pids: {17044}
2023-01-16T06:37:02.537 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-16T06:39:02.602 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...