2023-01-17T11:31:55.569 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-17T11:31:55.707 INFO:teuthology.lock.ops:Start node 'smithi145.front.sepia.ceph.com' reimaging 2023-01-17T11:31:55.708 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: reset os type and version on server 2023-01-17T11:31:55.708 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2023-01-17T11:31:55.723 INFO:teuthology.lock.ops:Node 'smithi145.front.sepia.ceph.com' reimaging is complete 2023-01-17T11:31:55.723 INFO:teuthology.lock.ops:Start node 'smithi151.front.sepia.ceph.com' reimaging 2023-01-17T11:31:55.723 INFO:teuthology.lock.ops:Updating [smithi151.front.sepia.ceph.com]: reset os type and version on server 2023-01-17T11:31:55.724 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2023-01-17T11:31:55.740 INFO:teuthology.lock.ops:Node 'smithi151.front.sepia.ceph.com' reimaging is complete 2023-01-17T11:31:55.740 INFO:teuthology.lock.ops:Start node 'smithi187.front.sepia.ceph.com' reimaging 2023-01-17T11:31:55.740 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: reset os type and version on server 2023-01-17T11:31:55.740 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-01-17T11:31:55.756 INFO:teuthology.lock.ops:Node 'smithi187.front.sepia.ceph.com' reimaging is complete 2023-01-17T11:31:55.861 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of ubuntu 20.04 2023-01-17T11:31:55.886 INFO:teuthology.provision.fog.smithi151:Scheduling deploy of ubuntu 20.04 2023-01-17T11:31:55.906 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of ubuntu 20.04 2023-01-17T11:31:56.048 INFO:teuthology.orchestra.console:Power off smithi145 2023-01-17T11:31:56.065 INFO:teuthology.orchestra.console:Power off smithi151 2023-01-17T11:31:56.087 INFO:teuthology.orchestra.console:Power off smithi187 2023-01-17T11:32:04.444 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2023-01-17T11:32:04.545 INFO:teuthology.orchestra.console:Power on smithi145 2023-01-17T11:32:08.564 INFO:teuthology.orchestra.console:Power off for smithi151 completed 2023-01-17T11:32:08.569 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2023-01-17T11:32:08.665 INFO:teuthology.orchestra.console:Power on smithi151 2023-01-17T11:32:08.672 INFO:teuthology.orchestra.console:Power on smithi187 2023-01-17T11:32:08.799 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2023-01-17T11:32:08.900 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2023-01-17T11:32:12.926 INFO:teuthology.orchestra.console:Power on for smithi187 completed 2023-01-17T11:32:13.027 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish 2023-01-17T11:32:17.034 INFO:teuthology.orchestra.console:Power on for smithi151 completed 2023-01-17T11:32:17.135 INFO:teuthology.provision.fog.smithi151:Waiting for deploy to finish 2023-01-17T11:36:13.161 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-17T11:36:17.257 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-17T11:36:21.609 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-01-17T11:36:22.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-17T11:36:26.313 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-17T11:36:30.665 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-01-17T11:36:31.273 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-17T11:36:35.369 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-17T11:36:39.721 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-01-17T11:36:40.329 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.145 2023-01-17T11:36:44.430 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-17T11:36:48.777 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-01-17T11:36:53.513 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-01-17T11:36:57.833 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.151 2023-01-17T11:37:18.568 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-17T11:37:32.234 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-17T11:37:40.052 INFO:teuthology.provision.fog.smithi145:Node is ready 2023-01-17T11:37:40.059 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com 2023-01-17T11:37:40.066 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145 2023-01-17T11:37:40.189 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2023-01-17T11:37:53.496 INFO:teuthology.provision.fog.smithi187:Node is ready 2023-01-17T11:37:53.503 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com 2023-01-17T11:37:53.510 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187 2023-01-17T11:37:53.687 INFO:teuthology.provision.fog.smithi187:Deploy complete! 2023-01-17T11:38:03.850 WARNING:teuthology.provision.fog:timed out 2023-01-17T11:38:10.083 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-17T11:38:10.646 INFO:teuthology.provision.fog.smithi151:Node is ready 2023-01-17T11:38:10.652 INFO:teuthology.orchestra.run.smithi151.stdout:smithi151.front.sepia.ceph.com 2023-01-17T11:38:10.699 INFO:teuthology.orchestra.run.smithi151.stdout:172.21.15.151 smithi151.front.sepia.ceph.com smithi151 2023-01-17T11:38:10.827 INFO:teuthology.provision.fog.smithi151:Deploy complete! 2023-01-17T11:38:11.177 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com 2023-01-17T11:38:11.177 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-17T11:38:11.203 INFO:teuthology.lock.ops:Checking smithi151.front.sepia.ceph.com 2023-01-17T11:38:11.204 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-17T11:38:11.221 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com 2023-01-17T11:38:11.221 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-17T11:38:11.239 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server 2023-01-17T11:38:11.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T11:38:11.544 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2023-01-17T11:38:11.550 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Ubuntu" 2023-01-17T11:38:11.550 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:ID=ubuntu 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE=debian 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="20.04" 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_CODENAME=focal 2023-01-17T11:38:11.551 INFO:teuthology.orchestra.run.smithi145.stdout:UBUNTU_CODENAME=focal 2023-01-17T11:38:11.553 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2023-01-17T11:38:11.577 INFO:teuthology.lock.ops:Updating [smithi151.front.sepia.ceph.com]: set os type and version on server 2023-01-17T11:38:11.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T11:38:11.747 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2023-01-17T11:38:11.753 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="Ubuntu" 2023-01-17T11:38:11.753 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-17T11:38:11.753 INFO:teuthology.orchestra.run.smithi151.stdout:ID=ubuntu 2023-01-17T11:38:11.753 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE=debian 2023-01-17T11:38:11.753 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-17T11:38:11.753 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="20.04" 2023-01-17T11:38:11.754 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-17T11:38:11.754 INFO:teuthology.orchestra.run.smithi151.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-17T11:38:11.754 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-17T11:38:11.754 INFO:teuthology.orchestra.run.smithi151.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-17T11:38:11.754 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_CODENAME=focal 2023-01-17T11:38:11.754 INFO:teuthology.orchestra.run.smithi151.stdout:UBUNTU_CODENAME=focal 2023-01-17T11:38:11.754 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2023-01-17T11:38:11.774 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server 2023-01-17T11:38:11.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T11:38:12.028 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2023-01-17T11:38:12.076 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="Ubuntu" 2023-01-17T11:38:12.076 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:ID=ubuntu 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE=debian 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="20.04" 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_CODENAME=focal 2023-01-17T11:38:12.077 INFO:teuthology.orchestra.run.smithi187.stdout:UBUNTU_CODENAME=focal 2023-01-17T11:38:12.078 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-01-17T11:38:12.189 INFO:teuthology.dispatcher.supervisor:Running job 7143440 2023-01-17T11:38:12.189 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2023-01-17_05:00:02-smoke-main-distro-default-smithi/7143440 --name teuthology-2023-01-17_05:00:02-smoke-main-distro-default-smithi --description smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_cache_snaps}} -- /home/teuthworker/archive/teuthology-2023-01-17_05:00:02-smoke-main-distro-default-smithi/7143440/orig.config.yaml 2023-01-17T11:38:12.194 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2023-01-17_05:00:02-smoke-main-distro-default-smithi/7143440 2023-01-17T11:38:12.194 INFO:teuthology.dispatcher.supervisor:Job PID: 31129 2023-01-17T11:38:12.195 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-17T23:39:35.652 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:39:35.827 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:39:36.598 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:39:36.598 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:39:36.598 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:39:36.598 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:39:36.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:39:36.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:39:36.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T23:39:37.470 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:39:37.633 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:39:47.644 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-01-17T23:39:52.735 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-01-17T23:39:58.300 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-17T23:39:59.413 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-01-17T23:40:24.544 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-17T23:40:32.047 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-01-17T23:40:55.785 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-17T23:40:58.853 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-17T23:41:36.699 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:41:37.424 INFO:teuthology.kill:Nuking machines: ['smithi145', 'smithi151', 'smithi187'] 2023-01-17T23:41:38.137 INFO:teuthology.kill:2023-01-17 23:41:38,137.137 INFO:teuthology.nuke:targets: 2023-01-17T23:41:38.205 INFO:teuthology.kill: smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-17T23:41:38.205 INFO:teuthology.kill: smithi151.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-17T23:41:38.205 INFO:teuthology.kill: smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-17T23:41:38.205 INFO:teuthology.kill:2023-01-17 23:41:38,137.137 INFO:teuthology.nuke:Checking targets against current locks 2023-01-17T23:41:38.205 INFO:teuthology.kill:2023-01-17 23:41:38,187.187 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-17T23:41:38.212 INFO:teuthology.kill:2023-01-17 23:41:38,212.212 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-17T23:41:38.231 INFO:teuthology.kill:2023-01-17 23:41:38,231.231 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-17T23:41:38.264 INFO:teuthology.kill:2023-01-17 23:41:38,264.264 INFO:teuthology.orchestra.console:Power off smithi145 2023-01-17T23:41:38.372 INFO:teuthology.kill:2023-01-17 23:41:38,300.300 INFO:teuthology.orchestra.console:Power off smithi151 2023-01-17T23:41:38.372 INFO:teuthology.kill:2023-01-17 23:41:38,332.332 INFO:teuthology.orchestra.console:Power off smithi187 2023-01-17T23:41:46.662 INFO:teuthology.kill:2023-01-17 23:41:46,661.661 INFO:teuthology.orchestra.console:Power off for smithi151 completed 2023-01-17T23:41:46.807 INFO:teuthology.kill:2023-01-17 23:41:46,807.807 INFO:teuthology.lock.ops:unlocked: smithi151.front.sepia.ceph.com 2023-01-17T23:41:50.750 INFO:teuthology.kill:2023-01-17 23:41:50,750.750 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2023-01-17T23:41:50.824 INFO:teuthology.kill:2023-01-17 23:41:50,805.805 INFO:teuthology.orchestra.console:Power off for smithi187 completed 2023-01-17T23:41:50.900 INFO:teuthology.kill:2023-01-17 23:41:50,900.900 INFO:teuthology.lock.ops:unlocked: smithi145.front.sepia.ceph.com 2023-01-17T23:41:50.947 INFO:teuthology.kill:2023-01-17 23:41:50,947.947 INFO:teuthology.lock.ops:unlocked: smithi187.front.sepia.ceph.com 2023-01-17T23:43:51.152 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:43:51.204 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:43:51.955 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:43:51.956 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:43:51.956 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:43:51.956 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:43:51.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:43:52.035 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:43:53.152 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:43:54.230 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:43:55.346 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:43:56.423 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:43:57.540 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:43:58.618 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:43:59.736 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:44:00.853 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:44:01.972 ERROR:teuthology.orchestra.connection:Error authenticating with smithi145.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:44:01.972 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi145 2023-01-17T23:44:02.030 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:44:02.766 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-17T23:46:02.817 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:46:02.872 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:46:03.695 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:46:03.695 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:46:03.695 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:46:03.696 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:46:03.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:46:19.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:46:19.743 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:46:51.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T23:46:51.820 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:46:51.894 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:46:51.928 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:46:51.929 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.021 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:46:52.021 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.059 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:46:52.059 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.060 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:46:52.084 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-17T23:46:52.089 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-17T23:46:52.123 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-17T23:46:52.149 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-17T23:46:52.149 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.176 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-17T23:46:52.176 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.200 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-17T23:46:52.200 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.201 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:46:52.224 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:46:52.243 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:46:52.264 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:46:52.289 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:46:52.289 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.316 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:46:52.316 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.340 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:46:52.340 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:46:52.393 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:46:53.097 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-17T23:48:53.151 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:48:53.230 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:48:53.996 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:48:53.996 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:48:53.996 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:48:53.996 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:48:53.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:48:54.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:48:54.155 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:48:54.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T23:48:54.335 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:48:54.505 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:48:54.530 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:48:54.530 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:48:54.557 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:48:54.558 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:48:54.580 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:48:54.581 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:48:54.582 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:48:54.791 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:48:54.813 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:48:54.820 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:48:54.839 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:48:54.878 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:48:54.879 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:48:54.905 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:48:54.906 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:48:54.929 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:48:54.929 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:48:54.980 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:48:55.721 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-17T23:50:55.776 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:50:55.831 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:50:56.587 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:50:56.588 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:50:56.588 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:50:56.588 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:50:56.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:50:56.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:50:56.788 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:50:56.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T23:50:56.965 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:50:57.136 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:50:57.161 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:50:57.161 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:50:57.188 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:50:57.188 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:50:57.214 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:50:57.214 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:50:57.215 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:50:57.407 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:50:57.430 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:50:57.435 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:50:57.469 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/ceph’: No such file or directory 2023-01-17T23:50:57.494 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:50:57.494 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:50:57.521 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:50:57.521 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:50:57.545 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-17T23:50:57.545 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:50:57.619 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:50:58.320 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-17T23:52:58.373 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:52:58.471 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:52:59.257 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:52:59.257 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:52:59.257 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:52:59.257 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:52:59.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:52:59.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:52:59.412 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:52:59.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T23:52:59.601 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:52:59.769 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:52:59.794 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:52:59.794 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:52:59.821 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:52:59.822 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:52:59.847 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:52:59.847 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:52:59.848 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:52:59.872 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-17T23:52:59.890 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-17T23:52:59.965 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:53:00.639 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-17T23:55:00.692 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:55:00.742 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:55:01.510 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:55:01.510 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:55:01.510 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:55:01.510 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:55:01.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:55:01.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:55:01.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T23:55:01.887 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:55:02.139 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:55:02.140 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:55:02.167 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:55:02.192 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-17T23:55:02.209 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-17T23:55:02.261 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:55:03.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-17T23:57:03.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:57:03.120 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:57:03.887 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:57:03.887 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:57:03.887 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:57:03.887 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:57:03.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:57:04.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:57:04.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-17T23:57:04.277 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-17T23:57:04.523 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-17T23:57:04.552 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:57:04.580 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-17T23:57:04.590 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory 2023-01-17T23:57:04.594 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-17T23:57:04.645 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:57:05.375 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-17T23:59:05.428 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-17T23:59:05.478 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:59:06.191 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-17T23:59:06.191 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-17T23:59:06.191 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-17T23:59:06.191 INFO:teuthology.misc:Compressing logs... 2023-01-17T23:59:06.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-17T23:59:06.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-17T23:59:37.946 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:39.061 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:40.176 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:41.292 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:42.407 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:43.523 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:44.638 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:45.753 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:46.868 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:47.983 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-17T23:59:47.983 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-17T23:59:48.033 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-17T23:59:48.767 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:01:48.819 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:01:48.871 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:01:49.659 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:01:49.659 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:01:49.659 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:01:49.659 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:01:49.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:01:49.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:01:50.665 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T00:01:50.715 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:01:51.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:03:51.434 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:03:51.588 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:03:52.532 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:03:52.624 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:03:52.624 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:03:52.624 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:03:52.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:03:52.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:03:52.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:03:53.003 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-18T00:03:53.368 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:03:53.368 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:03:53.396 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:03:53.573 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:03:53.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:03:53.626 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:03:54.484 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:05:54.703 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:05:54.757 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:05:55.480 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:05:55.480 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:05:55.480 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:05:55.480 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:05:55.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:05:55.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:05:55.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:05:55.857 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-18T00:05:56.087 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:05:56.087 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:05:56.113 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:05:56.139 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:05:56.155 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:05:56.205 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:05:56.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:07:56.973 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:07:57.027 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:07:57.805 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:07:57.806 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:07:57.806 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:07:57.806 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:07:57.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:07:57.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:07:58.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:07:58.635 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-18T00:07:58.855 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:07:58.856 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:07:58.881 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:07:58.906 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:07:58.923 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:07:58.972 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:07:59.683 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:09:59.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:09:59.791 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:10:00.547 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:10:00.547 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:10:00.547 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:10:00.547 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:10:00.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:10:00.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:10:00.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:10:00.886 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-18T00:10:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:10:01.117 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:10:01.144 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:10:01.170 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:10:01.185 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:10:01.275 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:10:01.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:12:02.045 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:12:02.094 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:12:02.873 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:12:02.874 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:12:02.874 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:12:02.874 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:12:02.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:12:03.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:12:03.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:12:03.246 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory 2023-01-18T00:12:03.468 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:12:03.469 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:12:03.495 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:12:03.520 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:12:03.537 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:12:03.538 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:12:03.591 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:12:04.427 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:14:04.477 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:14:04.540 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:14:05.334 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:14:05.335 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:14:05.335 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:14:05.335 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:14:05.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:14:05.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:14:23.849 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T00:14:23.904 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:14:24.625 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:16:24.680 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:16:24.732 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:16:25.462 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:16:25.462 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:16:25.462 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:16:25.462 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:16:25.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:16:25.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:16:25.758 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:26.870 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:27.982 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:29.094 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:30.206 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:31.278 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:32.390 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:33.461 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:34.573 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:35.685 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:16:35.685 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T00:16:35.737 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:16:36.502 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:18:36.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:18:36.606 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:18:37.405 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:18:37.405 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:18:37.405 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:18:37.405 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:18:37.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:18:37.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:19:09.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:19:09.832 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:19:10.442 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:19:10.442 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:19:10.501 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:19:10.523 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T00:19:10.526 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:19:10.527 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:19:10.578 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:19:11.265 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:21:11.318 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:21:11.370 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:21:12.074 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:21:12.074 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:21:12.074 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:21:12.075 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:21:12.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:21:12.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:21:45.705 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T00:21:45.756 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:21:46.508 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:23:46.562 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:23:46.612 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:23:47.404 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:23:47.404 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:23:47.404 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:23:47.404 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:23:47.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:23:47.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:23:47.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:23:47.902 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:23:48.186 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:23:48.186 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:23:48.226 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:23:48.253 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:23:48.269 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:23:48.321 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:23:49.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:25:49.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:25:49.160 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:25:49.953 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:25:49.953 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:25:49.953 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:25:49.953 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:25:49.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:25:50.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:25:50.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:25:50.389 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:25:50.643 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:25:50.643 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:25:50.674 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:25:50.698 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:25:50.716 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:25:50.767 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:25:51.497 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:27:51.553 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:27:51.611 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:27:52.354 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:27:52.354 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:27:52.355 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:27:52.355 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:27:52.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:27:52.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:27:52.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:27:52.735 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:27:52.978 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:27:52.978 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:27:53.005 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:27:53.030 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:27:53.048 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:27:53.100 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:27:53.828 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:29:53.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:29:53.985 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:29:54.747 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:29:54.748 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:29:54.748 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:29:54.748 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:29:54.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:29:54.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:29:55.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:29:55.417 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:29:55.443 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:29:55.445 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:29:55.495 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:29:56.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:31:56.293 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:31:56.347 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:31:57.101 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:31:57.102 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:31:57.102 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:31:57.102 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:31:57.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:31:57.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:31:57.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:31:57.813 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:31:57.855 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:31:57.857 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:31:57.910 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:31:58.613 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:33:58.665 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:33:58.726 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:33:59.470 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:33:59.470 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:33:59.470 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:33:59.471 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:33:59.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:33:59.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:33:59.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:34:00.097 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:34:00.126 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:34:00.139 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:34:00.187 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:34:00.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:36:00.957 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:36:01.010 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:36:01.788 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:36:01.789 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:36:01.789 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:36:01.789 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:36:01.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:36:01.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:36:20.457 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T00:36:20.507 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:36:21.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:38:21.244 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:38:21.295 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:38:22.123 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:38:22.123 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:38:22.124 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:38:22.124 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:38:22.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:38:22.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:38:22.377 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:23.449 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:24.558 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:25.670 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:26.783 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:27.893 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:28.964 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:30.076 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:31.186 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:32.304 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T00:38:32.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T00:38:32.355 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:38:33.072 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:40:33.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:40:33.182 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:40:33.926 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:40:33.974 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:40:33.974 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:40:33.974 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:40:33.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:40:34.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:40:34.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T00:40:34.630 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:40:35.480 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:42:35.528 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:42:35.700 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:42:36.610 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:42:36.611 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:42:36.611 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:42:36.611 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:42:36.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:42:36.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:42:36.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:42:36.969 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:42:37.235 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:42:37.235 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:42:37.266 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:42:37.283 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T00:42:37.295 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:42:37.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:42:37.361 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:42:38.104 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:44:38.158 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:44:38.208 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:44:38.967 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:44:38.968 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:44:38.968 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:44:38.968 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:44:38.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:44:39.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:44:40.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:44:41.001 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:44:41.089 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:44:41.089 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:44:41.121 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:44:41.151 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:44:41.152 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T00:44:41.164 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:44:41.222 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:44:41.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:46:42.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:46:42.076 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:46:42.926 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:46:42.926 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:46:42.926 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:46:42.926 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:46:42.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:46:43.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:46:43.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:46:43.322 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:46:43.557 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:46:43.558 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:46:43.589 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:46:43.617 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:46:43.631 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:46:43.682 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:46:44.387 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:48:44.438 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:48:44.494 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:48:45.297 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:48:45.297 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:48:45.297 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:48:45.297 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:48:45.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:48:45.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:48:45.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:48:45.664 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T00:48:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T00:48:45.882 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T00:48:45.911 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:48:45.940 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:48:45.954 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:48:46.006 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:48:46.704 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:50:46.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:50:46.811 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:50:47.631 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:50:47.631 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:50:47.631 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:50:47.631 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:50:47.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:50:47.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:50:47.976 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:50:48.256 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:50:48.286 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:50:48.299 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:50:48.350 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:50:49.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:52:49.125 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:52:49.179 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:52:49.928 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:52:49.928 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:52:49.928 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:52:49.928 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:52:49.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:52:50.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:52:50.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:52:50.503 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:52:50.532 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:52:50.545 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:52:50.594 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:52:51.319 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:54:51.372 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:54:51.419 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:54:52.193 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:54:52.193 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:54:52.194 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:54:52.194 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:54:52.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:54:52.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:54:52.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:54:52.839 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:54:52.868 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:54:52.881 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:54:52.949 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:54:53.652 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:56:53.704 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:56:53.778 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:56:54.572 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:56:54.572 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:56:54.572 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:56:54.572 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:56:54.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:56:54.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:56:54.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:56:55.215 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:56:55.243 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:56:55.258 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:56:55.309 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:56:56.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T00:58:56.092 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T00:58:56.140 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:58:56.856 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T00:58:56.856 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T00:58:56.856 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T00:58:56.856 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:58:56.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T00:58:57.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T00:58:57.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T00:58:57.428 INFO:teuthology.misc:Compressing logs... 2023-01-18T00:58:57.455 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T00:58:57.470 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T00:58:57.519 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T00:58:58.198 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:00:58.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:00:58.304 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:00:59.127 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:00:59.127 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:00:59.127 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:00:59.127 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:00:59.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:00:59.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:00:59.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:00:59.816 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:00:59.845 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:00:59.859 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:00:59.911 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:01:00.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:03:00.663 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:03:00.715 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:03:01.424 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:03:01.425 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:03:01.425 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:03:01.425 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:03:01.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:03:01.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:03:01.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:03:02.067 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:03:02.095 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:03:02.110 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:03:02.162 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:03:02.879 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:05:02.940 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:05:03.034 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:05:03.744 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:05:03.744 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:05:03.744 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:05:03.744 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:05:03.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:05:03.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:05:04.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:05:04.375 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:05:04.403 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:05:04.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:05:04.465 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:05:05.157 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:07:05.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:07:05.256 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:07:06.003 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:07:06.003 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:07:06.003 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:07:06.003 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:07:06.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:07:06.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:07:06.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:07:06.642 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:07:06.672 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:07:06.684 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:07:06.736 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:07:07.400 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:09:07.448 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:09:07.503 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:09:08.262 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:09:08.263 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:09:08.263 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:09:08.263 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:09:08.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:09:08.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:09:08.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:09:08.883 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:09:08.912 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:09:08.925 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:09:08.974 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:09:09.697 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:11:09.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:11:09.820 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:11:10.598 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:11:10.598 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:11:10.598 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:11:10.598 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:11:10.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:11:10.784 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:11:10.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:11:11.247 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:11:11.276 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:11:11.289 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:11:11.345 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:11:12.113 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:13:12.164 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:13:12.214 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:13:12.926 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:13:12.927 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:13:12.927 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:13:12.927 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:13:12.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:13:13.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:13:13.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:13:13.626 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:13:13.656 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:13:13.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:13:13.718 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:13:14.398 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:15:14.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:15:14.503 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:15:15.268 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:15:15.268 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:15:15.268 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:15:15.269 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:15:15.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:15:15.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:15:15.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:15:15.894 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:15:15.923 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:15:15.936 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:15:15.987 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:15:16.683 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:17:16.731 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:17:16.781 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:17:17.531 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:17:17.531 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:17:17.531 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:17:17.531 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:17:17.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:17:17.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:17:17.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:17:18.166 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:17:18.194 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:17:18.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:17:18.258 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:17:18.997 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:19:19.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:19:19.139 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:19:19.906 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:19:19.907 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:19:19.907 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:19:19.907 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:19:19.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:19:20.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:19:20.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:19:20.503 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:19:20.532 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:19:20.545 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:19:20.602 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:19:21.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:21:21.403 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:21:21.457 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:21:22.309 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:21:22.309 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:21:22.310 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:21:22.310 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:21:22.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:21:22.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:21:22.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:21:22.955 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:21:22.983 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:21:22.997 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:21:23.058 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:21:23.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:23:23.876 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:23:23.929 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:23:24.742 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:23:24.743 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:23:24.743 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:23:24.743 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:23:24.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:23:24.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:23:25.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:23:25.503 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:23:25.522 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T01:23:25.532 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:23:25.545 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:23:25.596 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:23:26.315 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:25:26.365 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:25:26.418 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:25:27.196 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:25:27.196 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:25:27.196 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:25:27.196 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:25:27.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:25:27.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:25:45.705 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T01:25:45.758 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:25:46.549 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:27:46.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:27:46.654 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:27:47.424 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:27:47.425 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:27:47.425 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:27:47.425 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:27:47.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:27:47.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:27:50.665 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T01:27:50.714 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:27:51.405 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:29:51.458 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:29:51.511 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:29:52.246 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:29:52.247 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:29:52.247 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:29:52.247 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:29:52.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:29:52.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:29:52.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:29:52.658 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:29:52.915 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:29:52.915 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:29:52.944 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:29:52.971 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T01:29:52.972 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:29:52.986 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:29:53.039 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:29:53.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:31:53.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:31:53.887 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:31:54.648 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:31:54.648 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:31:54.648 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:31:54.648 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:31:54.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:31:54.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:31:57.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T01:31:57.950 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:31:58.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:33:58.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:33:58.758 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:33:59.501 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:33:59.501 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:33:59.501 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:33:59.501 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:33:59.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:33:59.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:33:59.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:33:59.900 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:34:00.138 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:34:00.139 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:34:00.172 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:34:00.201 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:34:00.215 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:34:00.266 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:34:00.988 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:36:01.039 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:36:01.091 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:36:01.830 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:36:01.831 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:36:01.831 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:36:01.831 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:36:01.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:36:02.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:36:02.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:36:02.253 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:36:02.506 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:36:02.507 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:36:02.535 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:36:02.564 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:36:02.577 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:36:02.628 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:36:03.396 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:38:03.459 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:38:03.573 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:38:04.299 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:38:04.299 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:38:04.299 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:38:04.300 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:38:04.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:38:04.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:38:04.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:38:04.732 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:38:04.969 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:38:04.969 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:38:04.999 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:38:05.028 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:38:05.041 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:38:05.095 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:38:05.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:40:05.874 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:40:05.974 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:40:06.760 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:40:06.760 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:40:06.760 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:40:06.760 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:40:06.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:40:06.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:40:25.321 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T01:40:25.371 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:40:26.076 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:42:26.125 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:42:26.178 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:42:26.939 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:42:26.940 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:42:26.940 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:42:26.940 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:42:26.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:42:27.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:42:30.217 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T01:42:30.272 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:42:30.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:44:31.005 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:44:31.059 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:44:31.798 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:44:31.799 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:44:31.799 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:44:31.799 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:44:31.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:44:31.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:44:32.055 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:33.166 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:34.278 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:35.389 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:36.500 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:37.611 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:38.683 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:39.794 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:40.905 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:42.016 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T01:44:42.016 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T01:44:42.069 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:44:42.805 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:46:42.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:46:42.913 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:46:43.733 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:46:43.733 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:46:43.734 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:46:43.734 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:46:43.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:46:43.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:47:01.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:47:01.730 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:47:01.876 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:47:01.876 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:47:01.909 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:47:01.927 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T01:47:01.938 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:47:01.950 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:47:02.003 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:47:02.687 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:49:02.741 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:49:02.791 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:49:03.558 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:49:03.558 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:49:03.558 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:49:03.559 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:49:03.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:49:03.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:49:22.153 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T01:49:22.208 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:49:22.943 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:51:22.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:51:23.048 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:51:23.890 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:51:23.891 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:51:23.891 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:51:23.891 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:51:23.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:51:24.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:51:24.249 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:51:24.274 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:51:24.522 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:51:24.522 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:51:24.552 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:51:24.581 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:51:24.594 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:51:24.644 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:51:25.393 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:53:25.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:53:25.506 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:53:26.282 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:53:26.282 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:53:26.282 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:53:26.282 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:53:26.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:53:26.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:53:26.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:53:26.646 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:53:26.907 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:53:26.907 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:53:26.936 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:53:26.965 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:53:26.979 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:53:27.038 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:53:27.786 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:55:27.840 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:55:27.891 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:55:28.639 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:55:28.640 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:55:28.640 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:55:28.640 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:55:28.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:55:28.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:55:28.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:55:28.984 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T01:55:29.228 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T01:55:29.228 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T01:55:29.259 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:55:29.289 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:55:29.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:55:29.352 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:55:30.036 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:57:30.087 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:57:30.139 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:57:30.918 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:57:30.919 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:57:30.919 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:57:30.919 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:57:30.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:57:31.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:57:31.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:57:31.836 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:57:31.864 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:57:31.879 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:57:31.928 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:57:32.636 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T01:59:32.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T01:59:32.743 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:59:33.477 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T01:59:33.477 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T01:59:33.477 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T01:59:33.477 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:59:33.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T01:59:33.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T01:59:33.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T01:59:34.182 INFO:teuthology.misc:Compressing logs... 2023-01-18T01:59:34.211 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T01:59:34.212 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T01:59:34.225 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T01:59:34.277 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T01:59:35.002 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:01:35.054 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:01:35.108 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:01:35.886 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:01:35.887 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:01:35.887 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:01:35.887 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:01:35.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:01:36.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:01:54.537 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T02:01:54.589 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:01:55.296 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:03:55.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:03:55.405 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:03:56.119 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:03:56.120 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:03:56.120 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:03:56.120 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:03:56.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:03:56.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:03:56.385 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:03:57.504 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:03:58.621 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:03:59.698 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:04:00.812 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:04:01.925 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:04:03.040 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:04:04.118 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:04:05.234 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:04:06.349 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T02:04:06.349 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T02:04:06.401 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:04:07.122 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:06:07.178 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:06:07.229 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:06:07.966 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:06:07.966 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:06:07.966 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:06:07.966 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:06:07.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:06:08.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:06:09.673 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T02:06:09.724 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:06:10.430 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:08:10.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:08:10.539 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:08:11.288 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:08:11.288 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:08:11.288 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:08:11.289 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:08:11.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:08:11.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:08:11.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:08:11.645 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T02:08:11.910 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T02:08:11.911 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T02:08:11.941 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:08:11.971 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:08:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T02:08:11.983 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:08:12.035 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:08:12.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:10:12.809 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:10:12.863 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:10:13.619 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:10:13.620 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:10:13.620 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:10:13.620 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:10:13.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:10:13.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:10:15.849 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T02:10:15.904 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:10:16.674 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:12:16.740 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:12:16.790 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:12:17.549 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:12:17.550 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:12:17.550 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:12:17.550 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:12:17.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:12:17.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:12:17.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:12:17.916 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T02:12:18.152 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T02:12:18.153 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T02:12:18.183 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:12:18.211 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:12:18.225 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:12:18.293 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:12:19.017 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:14:19.077 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:14:19.138 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:14:19.887 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:14:19.887 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:14:19.887 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:14:19.888 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:14:19.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:14:20.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:14:20.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:14:20.254 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T02:14:20.471 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T02:14:20.471 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T02:14:20.500 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:14:20.529 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:14:20.543 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:14:20.632 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:14:21.397 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:16:21.447 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:16:21.499 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:16:22.232 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:16:22.233 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:16:22.233 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:16:22.233 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:16:22.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:16:22.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:16:22.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:16:22.878 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:16:22.906 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:16:22.921 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:16:22.975 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:16:23.706 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:18:23.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:18:23.817 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:18:24.616 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:18:24.616 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:18:24.616 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:18:24.616 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:18:24.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:18:24.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:18:24.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:18:25.211 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:18:25.240 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:18:25.254 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:18:25.307 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:18:26.012 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:20:26.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:20:26.120 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:20:26.863 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:20:26.863 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:20:26.863 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:20:26.863 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:20:26.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:20:27.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:20:27.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:20:27.507 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:20:27.536 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:20:27.550 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:20:27.601 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:20:28.345 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:22:28.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:22:28.448 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:22:29.219 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:22:29.219 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:22:29.220 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:22:29.220 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:22:29.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:22:29.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:22:29.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:22:29.851 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:22:29.880 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:22:29.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:22:29.945 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:22:30.670 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:24:30.722 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:24:30.776 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:24:31.515 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:24:31.515 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:24:31.515 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:24:31.515 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:24:31.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:24:31.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:24:31.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:24:32.189 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:24:32.217 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:24:32.230 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:24:32.280 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:24:32.992 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:26:33.064 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:26:33.113 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:26:33.818 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:26:33.819 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:26:33.819 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:26:33.819 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:26:33.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:26:33.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:26:34.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:26:34.467 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:26:34.496 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:26:34.510 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:26:34.558 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:26:35.241 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:28:35.293 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:28:35.361 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:28:36.127 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:28:36.127 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:28:36.127 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:28:36.128 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:28:36.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:28:36.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:28:36.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:28:36.760 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:28:36.788 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:28:36.803 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:28:36.855 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:28:37.588 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:30:37.640 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:30:37.691 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:30:38.455 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:30:38.455 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:30:38.455 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:30:38.455 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:30:38.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:30:38.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:30:38.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:30:39.087 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:30:39.115 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:30:39.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:30:39.180 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:30:39.882 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:32:39.936 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:32:39.987 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:32:40.773 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:32:40.773 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:32:40.774 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:32:40.774 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:32:40.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:32:40.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:32:41.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:32:41.397 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:32:41.425 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:32:41.439 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:32:41.491 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:32:42.207 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:34:42.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:34:42.310 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:34:43.074 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:34:43.074 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:34:43.074 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:34:43.074 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:34:43.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:34:43.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:34:43.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:34:43.752 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:34:43.780 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:34:43.793 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:34:43.843 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:34:44.552 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:36:44.605 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:36:44.669 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:36:45.479 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:36:45.479 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:36:45.479 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:36:45.479 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:36:45.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:36:45.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:36:45.840 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:36:46.127 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:36:46.156 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:36:46.170 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:36:46.221 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:36:46.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:38:46.992 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:38:47.043 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:38:47.816 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:38:47.816 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:38:47.816 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:38:47.816 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:38:47.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:38:47.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:38:48.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:38:48.475 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:38:48.504 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:38:48.517 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:38:48.569 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:38:49.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:40:49.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:40:49.343 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:40:50.054 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:40:50.054 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:40:50.054 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:40:50.054 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:40:50.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:40:50.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:40:50.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:40:50.694 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:40:50.723 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:40:50.736 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:40:50.787 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:40:51.464 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:42:51.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:42:51.567 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:42:52.327 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:42:52.328 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:42:52.328 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:42:52.328 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:42:52.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:42:52.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:42:52.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:42:52.963 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:42:52.991 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:42:53.005 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:42:53.057 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:42:53.753 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:44:53.808 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:44:53.860 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:44:54.581 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:44:54.581 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:44:54.581 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:44:54.581 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:44:54.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:44:54.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:44:54.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:44:55.235 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:44:55.265 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:44:55.278 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:44:55.330 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:44:56.035 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:46:56.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:46:56.144 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:46:56.875 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:46:56.875 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:46:56.875 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:46:56.875 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:46:56.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:46:57.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:46:57.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:46:57.516 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:46:57.546 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:46:57.558 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:46:57.620 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:46:58.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:48:58.358 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:48:58.408 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:48:59.153 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:48:59.153 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:48:59.153 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:48:59.153 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:48:59.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:48:59.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:48:59.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:48:59.790 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:48:59.818 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:48:59.832 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:48:59.882 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:49:00.610 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:51:00.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:51:00.764 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:51:01.477 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:51:01.477 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:51:01.477 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:51:01.477 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:51:01.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:51:01.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:51:01.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:51:02.046 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:51:02.074 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:51:02.088 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:51:02.142 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:51:02.850 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:53:02.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:53:02.949 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:53:03.696 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:53:03.696 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:53:03.696 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:53:03.696 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:53:03.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:53:03.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:53:04.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:53:04.329 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:53:04.358 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:53:04.371 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:53:04.420 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:53:05.123 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:55:05.178 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:55:05.232 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:55:06.005 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:55:06.005 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:55:06.005 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:55:06.005 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:55:06.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:55:06.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:55:06.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:55:06.644 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:55:06.673 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:55:06.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:55:06.735 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:55:07.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:57:07.466 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:57:07.521 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:57:08.243 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:57:08.243 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:57:08.243 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:57:08.243 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:57:08.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:57:08.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:57:08.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:57:08.904 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:57:08.932 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:57:08.946 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:57:08.996 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:57:09.665 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T02:59:09.721 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T02:59:09.778 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:59:10.550 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T02:59:10.551 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T02:59:10.551 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T02:59:10.551 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:59:10.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T02:59:10.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T02:59:10.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T02:59:11.155 INFO:teuthology.misc:Compressing logs... 2023-01-18T02:59:11.183 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T02:59:11.198 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T02:59:11.248 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T02:59:11.946 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:01:11.996 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:01:12.049 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:01:12.813 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:01:12.813 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:01:12.813 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:01:12.813 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:01:12.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:01:12.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:01:13.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:01:13.466 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:01:13.495 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:01:13.508 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:01:13.565 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:01:14.275 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:03:14.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:03:14.378 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:03:15.073 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:03:15.074 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:03:15.074 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:03:15.074 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:03:15.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:03:15.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:03:15.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:03:15.737 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:03:15.766 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:03:15.780 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:03:15.832 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:03:16.560 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:05:16.614 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:05:16.668 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:05:17.408 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:05:17.409 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:05:17.409 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:05:17.409 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:05:17.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:05:17.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:05:17.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:05:18.087 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:05:18.118 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:05:18.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:05:18.190 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:05:18.919 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:07:18.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:07:19.019 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:07:19.777 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:07:19.777 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:07:19.777 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:07:19.777 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:07:19.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:07:19.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:07:20.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:07:20.414 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:07:20.443 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:07:20.457 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:07:20.507 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:07:21.182 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:09:21.246 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:09:21.302 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:09:22.024 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:09:22.025 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:09:22.025 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:09:22.025 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:09:22.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:09:22.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:09:22.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:09:22.681 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:09:22.710 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:09:22.723 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:09:22.771 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:09:23.429 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:11:23.487 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:11:23.539 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:11:24.273 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:11:24.273 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:11:24.273 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:11:24.273 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:11:24.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:11:24.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:11:24.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:11:24.901 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:11:24.926 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:11:24.943 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:11:24.994 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:11:25.716 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:13:25.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:13:25.816 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:13:26.598 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:13:26.598 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:13:26.598 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:13:26.599 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:13:26.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:13:26.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:13:26.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:13:27.253 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:13:27.281 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:13:27.282 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:13:27.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:13:27.348 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:13:28.039 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:15:28.095 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:15:28.145 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:15:28.840 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:15:28.840 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:15:28.841 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:15:28.841 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:15:28.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:15:29.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:15:29.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:15:29.495 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:15:29.524 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:15:29.524 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:15:29.537 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:15:29.586 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:15:30.244 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:17:30.301 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:17:30.368 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:17:31.116 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:17:31.117 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:17:31.117 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:17:31.117 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:17:31.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:17:31.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:17:31.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:17:31.719 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:17:31.748 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:17:31.748 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:17:31.761 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:17:31.812 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:17:32.526 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:19:32.578 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:19:32.630 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:19:33.405 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:19:33.405 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:19:33.405 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:19:33.405 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:19:33.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:19:33.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:19:33.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:19:34.055 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:19:34.082 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:19:34.083 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:19:34.097 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:19:34.148 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:19:34.936 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:21:34.996 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:21:35.048 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:21:35.769 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:21:35.769 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:21:35.770 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:21:35.770 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:21:35.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:21:35.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:21:36.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:21:36.449 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:21:36.477 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:21:36.477 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:21:36.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:21:36.538 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:21:37.228 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:23:37.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:23:37.332 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:23:38.061 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:23:38.061 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:23:38.061 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:23:38.061 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:23:38.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:23:38.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:23:38.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:23:38.748 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:23:38.778 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:23:38.778 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:23:38.790 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:23:38.847 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:23:39.570 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:25:39.620 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:25:39.710 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:25:40.448 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:25:40.448 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:25:40.448 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:25:40.448 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:25:40.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:25:40.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:25:40.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:25:41.100 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:25:41.128 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:25:41.129 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:25:41.142 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:25:41.195 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:25:41.935 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:27:41.984 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:27:42.044 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:27:42.753 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:27:42.753 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:27:42.753 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:27:42.753 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:27:42.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:27:42.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:27:43.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:27:43.417 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:27:43.448 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:27:43.448 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:27:43.459 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:27:43.513 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:27:44.222 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:29:44.272 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:29:44.323 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:29:45.042 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:29:45.042 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:29:45.042 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:29:45.042 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:29:45.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:29:45.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:29:45.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:29:45.676 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:29:45.704 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:29:45.704 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:29:45.718 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:29:45.768 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:29:46.435 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:31:46.487 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:31:46.540 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:31:47.300 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:31:47.300 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:31:47.301 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:31:47.301 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:31:47.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:31:47.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:31:47.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:31:47.974 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:31:48.004 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:31:48.004 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:31:48.017 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:31:48.067 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:31:48.787 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:33:48.843 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:33:48.894 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:33:49.636 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:33:49.636 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:33:49.636 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:33:49.636 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:33:49.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:33:49.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:33:49.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:33:50.293 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:33:50.321 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:33:50.321 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:33:50.335 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:33:50.389 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:33:51.162 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:35:51.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:35:51.274 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:35:51.987 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:35:51.988 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:35:51.988 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:35:51.988 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:35:51.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:35:52.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:35:52.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:35:52.626 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:35:52.656 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:35:52.656 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:35:52.668 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:35:52.718 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:35:53.409 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:37:53.464 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:37:53.515 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:37:54.221 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:37:54.221 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:37:54.221 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:37:54.221 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:37:54.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:37:54.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:37:54.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:37:54.856 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:37:54.885 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:37:54.885 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:37:54.898 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:37:54.949 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:37:55.631 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:39:55.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:39:55.738 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:39:56.496 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:39:56.496 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:39:56.496 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:39:56.496 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:39:56.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:39:56.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:39:56.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:39:57.154 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:39:57.183 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:39:57.183 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:39:57.197 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:39:57.249 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:39:57.948 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:41:57.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:41:58.050 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:41:58.854 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:41:58.855 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:41:58.855 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:41:58.855 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:41:58.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:41:59.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:41:59.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:41:59.535 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:41:59.564 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:41:59.564 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:41:59.577 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:41:59.642 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:42:00.376 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:44:00.428 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:44:00.479 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:44:01.214 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:44:01.214 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:44:01.214 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:44:01.214 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:44:01.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:44:01.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:44:01.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:44:01.851 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:44:01.880 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:44:01.880 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:44:01.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:44:01.942 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:44:02.617 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:46:02.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:46:02.724 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:46:03.472 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:46:03.472 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:46:03.472 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:46:03.472 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:46:03.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:46:03.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:46:03.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:46:04.136 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:46:04.166 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:46:04.166 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:46:04.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:46:04.230 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:46:04.957 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:48:05.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:48:05.062 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:48:05.823 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:48:05.823 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:48:05.823 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:48:05.823 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:48:05.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:48:06.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:48:06.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:48:06.464 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:48:06.492 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:48:06.492 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:48:06.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:48:06.557 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:48:07.280 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:50:07.328 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:50:07.381 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:50:08.106 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:50:08.106 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:50:08.107 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:50:08.107 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:50:08.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:50:08.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:50:08.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:50:08.710 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:50:08.738 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:50:08.738 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:50:08.752 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:50:08.815 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:50:09.504 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:52:09.558 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:52:09.609 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:52:10.308 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:52:10.308 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:52:10.309 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:52:10.309 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:52:10.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:52:10.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:52:10.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:52:10.921 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:52:10.949 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:52:10.950 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:52:10.963 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:52:11.013 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:52:11.712 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:54:11.764 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:54:11.815 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:54:12.553 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:54:12.553 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:54:12.553 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:54:12.554 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:54:12.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:54:12.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:54:12.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:54:13.196 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:54:13.224 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:54:13.225 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:54:13.239 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:54:13.289 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:54:13.986 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:56:14.049 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:56:14.099 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:56:14.818 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:56:14.818 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:56:14.818 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:56:14.818 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:56:14.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:56:14.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:56:15.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:56:15.458 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:56:15.487 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:56:15.487 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:56:15.500 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:56:15.551 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:56:16.229 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T03:58:16.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T03:58:16.352 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:58:17.077 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T03:58:17.077 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T03:58:17.077 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T03:58:17.078 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:58:17.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T03:58:17.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T03:58:17.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T03:58:17.733 INFO:teuthology.misc:Compressing logs... 2023-01-18T03:58:17.761 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T03:58:17.761 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T03:58:17.776 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T03:58:17.822 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T03:58:18.500 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:00:18.554 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:00:18.633 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:00:19.377 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:00:19.377 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:00:19.377 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:00:19.377 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:00:19.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:00:19.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:00:19.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:00:20.016 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:00:20.045 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:00:20.045 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:00:20.058 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:00:20.112 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:00:20.811 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:02:20.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:02:20.938 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:02:21.726 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:02:21.726 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:02:21.727 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:02:21.727 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:02:21.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:02:21.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:02:22.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:02:22.371 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:02:22.400 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:02:22.400 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:02:22.414 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:02:22.463 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:02:23.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:04:23.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:04:23.326 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:04:24.034 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:04:24.035 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:04:24.035 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:04:24.035 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:04:24.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:04:24.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:04:24.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:04:24.697 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:04:24.724 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:04:24.724 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:04:24.739 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:04:24.784 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:04:25.460 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:06:25.520 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:06:25.580 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:06:26.333 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:06:26.334 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:06:26.334 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:06:26.334 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:06:26.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:06:26.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:06:26.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:06:26.979 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:06:27.010 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:06:27.011 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:06:27.021 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:06:27.072 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:06:27.767 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:08:27.821 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:08:27.873 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:08:28.609 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:08:28.610 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:08:28.610 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:08:28.610 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:08:28.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:08:28.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:08:28.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:08:29.253 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:08:29.282 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:08:29.282 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:08:29.296 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:08:29.378 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:08:30.113 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:10:30.175 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:10:30.225 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:10:30.957 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:10:30.957 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:10:30.957 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:10:30.957 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:10:30.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:10:31.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:10:31.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:10:31.599 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:10:31.627 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:10:31.628 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:10:31.641 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:10:31.692 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:10:32.381 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:12:32.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:12:32.482 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:12:33.208 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:12:33.209 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:12:33.209 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:12:33.209 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:12:33.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:12:33.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:12:33.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:12:33.851 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:12:33.880 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:12:33.880 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:12:33.894 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:12:33.944 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:12:34.611 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:14:34.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:14:34.713 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:14:35.410 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:14:35.410 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:14:35.410 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:14:35.411 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:14:35.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:14:35.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:14:35.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:14:36.027 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:14:36.054 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:14:36.054 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:14:36.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:14:36.120 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:14:36.815 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:16:36.866 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:16:36.918 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:16:37.696 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:16:37.697 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:16:37.697 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:16:37.697 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:16:37.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:16:37.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:16:38.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:16:38.346 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:16:38.375 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:16:38.375 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:16:38.388 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:16:38.440 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:16:39.139 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:18:39.189 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:18:39.241 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:18:39.999 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:18:39.999 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:18:39.999 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:18:39.999 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:18:39.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:18:40.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:18:40.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:18:40.643 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:18:40.673 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:18:40.674 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:18:40.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:18:40.747 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:18:41.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:20:41.565 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:20:41.621 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:20:42.343 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:20:42.343 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:20:42.343 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:20:42.343 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:20:42.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:20:42.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:20:42.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:20:42.935 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:20:42.962 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:20:42.963 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:20:42.978 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:20:43.030 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:20:43.736 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:22:43.794 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:22:43.843 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:22:44.534 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:22:44.534 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:22:44.534 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:22:44.534 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:22:44.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:22:44.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:22:44.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:22:45.190 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:22:45.219 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:22:45.219 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:22:45.233 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:22:45.282 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:22:45.983 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:24:46.055 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:24:46.114 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:24:46.868 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:24:46.869 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:24:46.869 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:24:46.869 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:24:46.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:24:47.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:24:47.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:24:47.500 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:24:47.529 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:24:47.529 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:24:47.543 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:24:47.593 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:24:48.283 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:26:48.344 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:26:48.392 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:26:49.119 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:26:49.119 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:26:49.119 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:26:49.119 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:26:49.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:26:49.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:26:49.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:26:49.744 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:26:49.772 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:26:49.772 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:26:49.786 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:26:49.836 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:26:50.554 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:28:50.612 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:28:50.664 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:28:51.371 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:28:51.371 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:28:51.372 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:28:51.372 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:28:51.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:28:51.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:28:51.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:28:52.016 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:28:52.045 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:28:52.046 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:28:52.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:28:52.107 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:28:52.771 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:30:52.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:30:52.875 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:30:53.641 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:30:53.642 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:30:53.642 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:30:53.642 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:30:53.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:30:53.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:30:53.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:30:54.279 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:30:54.307 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:30:54.307 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:30:54.321 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:30:54.370 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:30:55.059 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:32:55.110 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:32:55.168 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:32:55.886 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:32:55.886 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:32:55.886 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:32:55.887 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:32:55.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:32:56.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:32:56.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:32:56.522 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:32:56.548 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:32:56.549 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:32:56.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:32:56.614 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:32:57.311 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:34:57.366 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:34:57.466 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:34:58.191 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:34:58.191 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:34:58.192 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:34:58.192 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:34:58.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:34:58.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:34:58.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:34:58.842 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:34:58.872 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:34:58.873 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:34:58.884 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:34:58.933 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:34:59.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:36:59.666 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:36:59.718 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:37:00.475 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:37:00.476 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:37:00.476 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:37:00.476 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:37:00.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:37:00.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:37:00.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:37:01.113 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:37:01.141 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:37:01.142 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:37:01.156 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:37:01.207 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:37:01.904 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:39:01.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:39:02.005 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:39:02.809 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:39:02.810 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:39:02.810 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:39:02.810 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:39:02.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:39:02.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:39:03.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:39:03.499 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:39:03.528 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:39:03.528 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:39:03.541 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:39:03.590 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:39:04.316 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:41:04.371 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:41:04.420 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:41:05.125 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:41:05.125 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:41:05.125 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:41:05.125 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:41:05.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:41:05.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:41:05.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:41:05.825 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:41:05.853 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:41:05.853 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:41:05.868 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:41:05.917 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:41:06.593 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:43:06.644 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:43:06.694 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:43:07.431 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:43:07.432 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:43:07.432 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:43:07.432 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:43:07.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:43:07.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:43:07.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:43:08.028 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:43:08.057 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:43:08.057 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:43:08.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:43:08.118 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:43:08.798 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:45:08.849 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:45:08.899 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:45:09.642 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:45:09.642 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:45:09.642 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:45:09.642 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:45:09.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:45:09.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:45:09.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:45:10.234 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:45:10.262 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:45:10.263 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:45:10.276 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:45:10.326 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:45:11.005 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:47:11.058 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:47:11.109 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:47:11.832 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:47:11.832 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:47:11.832 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:47:11.833 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:47:11.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:47:11.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:47:12.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:47:12.526 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:47:12.555 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:47:12.555 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:47:12.556 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T04:47:12.569 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:47:12.618 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:47:13.307 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:49:13.365 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:49:13.417 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:49:14.198 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:49:14.198 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:49:14.198 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:49:14.198 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:49:14.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:49:14.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:49:32.777 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T04:49:32.852 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:49:33.600 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:51:33.651 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:51:33.714 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:51:34.486 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:51:34.486 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:51:34.486 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:51:34.487 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:51:34.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:51:34.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:51:34.788 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:51:35.903 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:51:36.979 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:51:38.056 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:51:39.172 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:51:40.248 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:51:41.322 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:51:42.398 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T04:52:43.399 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T04:52:43.450 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:52:44.169 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:54:44.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:54:44.275 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:54:44.996 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:54:44.996 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:54:44.996 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:54:44.996 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:54:44.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:54:45.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:54:45.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:54:45.398 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T04:54:45.679 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T04:54:45.679 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T04:54:45.708 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:54:45.728 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T04:54:45.740 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:54:45.740 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:54:45.750 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:54:45.806 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:54:46.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:56:46.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:56:46.635 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:56:47.381 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:56:47.381 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:56:47.381 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:56:47.382 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:56:47.382 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:56:47.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:56:48.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:56:48.268 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T04:56:48.463 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T04:56:48.464 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T04:56:48.494 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:56:48.523 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:56:48.523 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:56:48.524 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T04:56:48.536 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:56:48.588 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:56:49.308 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T04:58:49.362 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T04:58:49.414 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:58:50.154 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T04:58:50.154 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T04:58:50.155 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T04:58:50.155 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:58:50.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T04:58:50.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T04:58:50.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T04:58:50.472 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T04:58:50.715 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T04:58:50.716 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T04:58:50.744 INFO:teuthology.misc:Compressing logs... 2023-01-18T04:58:50.774 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T04:58:50.774 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T04:58:50.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T04:58:50.843 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T04:58:51.537 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:00:51.591 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:00:51.641 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:00:52.359 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:00:52.359 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:00:52.359 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:00:52.359 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:00:52.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:00:52.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:00:52.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:00:52.725 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T05:00:52.979 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T05:00:52.980 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T05:00:53.009 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:00:53.038 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:00:53.038 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:00:53.051 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:00:53.103 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:00:53.782 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:02:53.840 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:02:53.891 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:02:54.610 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:02:54.610 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:02:54.610 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:02:54.611 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:02:54.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:02:54.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:02:54.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:02:55.247 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:02:55.276 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:02:55.277 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:02:55.290 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:02:55.339 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:02:56.040 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:04:56.095 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:04:56.142 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:04:56.891 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:04:56.891 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:04:56.891 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:04:56.891 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:04:56.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:04:57.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:04:57.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:04:57.524 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:04:57.552 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:04:57.552 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:04:57.566 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:04:57.619 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:04:58.290 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:06:58.342 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:06:58.396 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:06:59.163 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:06:59.163 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:06:59.163 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:06:59.163 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:06:59.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:06:59.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:06:59.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:06:59.752 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:06:59.780 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:06:59.780 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:06:59.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:06:59.839 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:07:00.513 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:09:00.580 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:09:00.628 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:09:01.388 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:09:01.388 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:09:01.388 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:09:01.388 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:09:01.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:09:01.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:09:01.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:09:01.963 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:09:01.991 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:09:01.991 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:09:02.005 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:09:02.091 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:09:02.779 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:11:02.827 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:11:02.879 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:11:03.591 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:11:03.591 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:11:03.591 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:11:03.592 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:11:03.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:11:03.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:11:03.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:11:04.260 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:11:04.289 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:11:04.289 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:11:04.303 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:11:04.352 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:11:05.019 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:13:05.073 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:13:05.126 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:13:05.897 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:13:05.897 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:13:05.898 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:13:05.898 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:13:05.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:13:06.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:13:06.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:13:06.544 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:13:06.573 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:13:06.574 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:13:06.587 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:13:06.640 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:13:07.359 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:15:07.427 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:15:07.504 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:15:08.255 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:15:08.255 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:15:08.255 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:15:08.256 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:15:08.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:15:08.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:15:08.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:15:08.879 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:15:08.908 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:15:08.908 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:15:08.921 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:15:08.973 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:15:09.702 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:17:09.756 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:17:09.806 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:17:10.522 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:17:10.523 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:17:10.523 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:17:10.523 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:17:10.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:17:10.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:17:10.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:17:11.147 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:17:11.175 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:17:11.176 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:17:11.188 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:17:11.238 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:17:11.934 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:19:11.981 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:19:12.033 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:19:12.750 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:19:12.750 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:19:12.750 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:19:12.750 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:19:12.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:19:12.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:19:13.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:19:13.409 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:19:13.437 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:19:13.438 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:19:13.452 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:19:13.501 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:19:14.236 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:21:14.284 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:21:14.334 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:21:15.061 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:21:15.062 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:21:15.062 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:21:15.062 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:21:15.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:21:15.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:21:15.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:21:15.631 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:21:15.658 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:21:15.658 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:21:15.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:21:15.723 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:21:16.386 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:23:16.439 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:23:16.493 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:23:17.305 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:23:17.305 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:23:17.305 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:23:17.305 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:23:17.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:23:17.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:23:17.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:23:17.883 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:23:17.912 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:23:17.912 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:23:17.925 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:23:17.977 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:23:18.669 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:25:18.724 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:25:18.773 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:25:19.475 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:25:19.476 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:25:19.476 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:25:19.476 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:25:19.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:25:19.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:25:19.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:25:20.123 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:25:20.151 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:25:20.151 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:25:20.165 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:25:20.216 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:25:20.907 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:27:20.955 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:27:21.005 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:27:21.772 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:27:21.772 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:27:21.772 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:27:21.772 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:27:21.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:27:21.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:27:22.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:27:22.393 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:27:22.421 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:27:22.421 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:27:22.435 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:27:22.487 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:27:23.195 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:29:23.253 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:29:23.301 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:29:24.020 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:29:24.020 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:29:24.021 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:29:24.021 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:29:24.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:29:24.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:29:24.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:29:24.632 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:29:24.661 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:29:24.661 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:29:24.674 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:29:24.725 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:29:25.451 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:31:25.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:31:25.569 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:31:26.357 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:31:26.357 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:31:26.357 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:31:26.357 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:31:26.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:31:26.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:31:26.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:31:27.012 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:31:27.042 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:31:27.042 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:31:27.054 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:31:27.104 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:31:27.821 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:33:27.875 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:33:27.931 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:33:28.674 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:33:28.675 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:33:28.675 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:33:28.675 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:33:28.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:33:28.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:33:29.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:33:29.293 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:33:29.320 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:33:29.320 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:33:29.335 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:33:29.387 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:33:30.100 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:35:30.153 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:35:30.212 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:35:30.947 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:35:30.947 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:35:30.948 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:35:30.948 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:35:30.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:35:31.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:35:31.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:35:31.528 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:35:31.557 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:35:31.557 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:35:31.571 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:35:31.621 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:35:32.314 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:37:32.367 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:37:32.417 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:37:33.138 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:37:33.138 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:37:33.138 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:37:33.138 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:37:33.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:37:33.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:37:33.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:37:33.785 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:37:33.813 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:37:33.813 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:37:33.827 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:37:33.878 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:37:34.563 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:39:34.623 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:39:34.686 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:39:35.449 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:39:35.450 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:39:35.450 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:39:35.450 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:39:35.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:39:35.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:39:35.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:39:36.059 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:39:36.088 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:39:36.088 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:39:36.101 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:39:36.156 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:39:36.901 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:41:36.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:41:37.019 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:41:37.759 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:41:37.759 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:41:37.760 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:41:37.760 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:41:37.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:41:37.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:41:38.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:41:38.390 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:41:38.419 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:41:38.419 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:41:38.432 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:41:38.482 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:41:39.216 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:43:39.271 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:43:39.337 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:43:40.088 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:43:40.088 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:43:40.088 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:43:40.088 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:43:40.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:43:40.225 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:43:40.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:43:40.685 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:43:40.714 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:43:40.714 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:43:40.727 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:43:40.819 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:43:41.497 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:45:41.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:45:41.599 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:45:42.320 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:45:42.321 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:45:42.321 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:45:42.321 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:45:42.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:45:42.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:45:42.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:45:42.947 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:45:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:45:42.977 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:45:42.989 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:45:43.039 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:45:43.738 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:47:43.809 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:47:43.859 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:47:44.623 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:47:44.623 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:47:44.623 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:47:44.623 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:47:44.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:47:44.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:47:44.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:47:45.246 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:47:45.276 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:47:45.276 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:47:45.289 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:47:45.339 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:47:46.030 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:49:46.080 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:49:46.135 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:49:46.898 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:49:46.898 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:49:46.898 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:49:46.898 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:49:46.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:49:47.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:49:47.251 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:49:47.555 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:49:47.584 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:49:47.585 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:49:47.598 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:49:47.651 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:49:48.367 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:51:48.422 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:51:48.473 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:51:49.195 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:51:49.196 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:51:49.196 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:51:49.196 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:51:49.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:51:49.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:51:49.534 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:51:49.815 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:51:49.844 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:51:49.845 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:51:49.858 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:51:49.909 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:51:50.681 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:53:50.731 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:53:50.783 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:53:51.518 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:53:51.518 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:53:51.518 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:53:51.518 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:53:51.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:53:51.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:53:51.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:53:52.101 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:53:52.131 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:53:52.132 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:53:52.143 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:53:52.192 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:53:52.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:55:52.901 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:55:52.952 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:55:53.657 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:55:53.658 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:55:53.658 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:55:53.658 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:55:53.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:55:53.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:55:54.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:55:54.308 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:55:54.337 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:55:54.337 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:55:54.350 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:55:54.401 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:55:55.116 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:57:55.170 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:57:55.222 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:57:55.935 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:57:55.935 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:57:55.935 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:57:55.935 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:57:55.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:57:56.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:57:56.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:57:56.587 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:57:56.617 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:57:56.617 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:57:56.630 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:57:56.682 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:57:57.362 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T05:59:57.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T05:59:57.466 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:59:58.235 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T05:59:58.235 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T05:59:58.235 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T05:59:58.236 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:59:58.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T05:59:58.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T05:59:58.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T05:59:58.849 INFO:teuthology.misc:Compressing logs... 2023-01-18T05:59:58.879 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T05:59:58.879 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T05:59:58.892 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T05:59:58.941 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T05:59:59.646 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:01:59.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:01:59.749 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:02:00.553 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:02:00.553 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:02:00.553 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:02:00.553 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:02:00.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:02:00.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:02:00.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:02:01.216 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:02:01.245 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:02:01.245 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:02:01.258 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:02:01.310 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:02:02.008 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:04:02.059 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:04:02.110 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:04:02.817 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:04:02.818 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:04:02.818 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:04:02.818 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:04:02.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:04:03.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:04:03.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:04:03.487 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:04:03.516 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:04:03.516 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:04:03.529 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:04:03.577 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:04:04.253 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:06:04.305 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:06:04.355 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:06:05.066 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:06:05.067 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:06:05.067 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:06:05.067 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:06:05.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:06:05.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:06:05.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:06:05.686 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:06:05.715 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:06:05.716 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:06:05.728 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:06:05.778 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:06:06.442 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:08:06.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:08:06.551 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:08:07.259 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:08:07.259 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:08:07.259 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:08:07.259 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:08:07.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:08:07.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:08:07.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:08:08.005 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:08:08.033 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:08:08.033 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:08:08.036 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T06:08:08.047 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:08:08.101 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:08:08.812 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:10:08.866 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:10:08.918 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:10:09.626 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:10:09.626 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:10:09.626 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:10:09.626 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:10:09.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:10:09.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:10:28.329 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T06:10:28.379 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:10:29.060 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:12:29.113 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:12:29.164 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:12:29.921 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:12:29.921 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:12:29.922 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:12:29.922 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:12:29.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:12:30.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:12:30.180 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:31.254 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:32.327 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:33.438 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:34.549 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:35.661 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:36.733 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:37.845 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:38.957 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:40.067 ERROR:teuthology.orchestra.connection:Error authenticating with smithi151.front.sepia.ceph.com: Authentication failed. 2023-01-18T06:12:40.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T06:12:40.118 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:12:40.780 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:14:40.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:14:40.887 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:14:41.640 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:14:41.641 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:14:41.641 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:14:41.641 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:14:41.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:14:41.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:14:44.877 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T06:14:44.951 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:14:45.683 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:16:45.751 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:16:45.805 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:16:46.514 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:16:46.514 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:16:46.514 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:16:46.514 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:16:46.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:16:46.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:16:46.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:16:46.838 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T06:16:47.053 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T06:16:47.054 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T06:16:47.083 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:16:47.111 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:16:47.111 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:16:47.113 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-18T06:16:47.125 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:16:47.179 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:16:47.865 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:18:48.044 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:18:48.093 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:18:48.815 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:18:48.816 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:18:48.816 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:18:48.816 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:18:48.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:18:48.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:18:50.871 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi151 2023-01-18T06:18:50.958 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:18:51.648 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:20:51.706 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:20:51.760 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:20:52.600 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:20:52.601 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:20:52.601 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:20:52.601 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:20:52.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:20:52.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:20:52.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:20:52.930 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T06:20:53.169 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T06:20:53.169 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T06:20:53.199 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:20:53.227 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:20:53.227 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:20:53.235 INFO:teuthology.orchestra.run.smithi151.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-01-18T06:20:53.241 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:20:53.296 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:20:54.027 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:22:54.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:22:54.143 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:22:54.897 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:22:54.897 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:22:54.897 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:22:54.897 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:22:54.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:22:55.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:22:55.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:22:55.211 INFO:teuthology.orchestra.run.smithi151.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-18T06:22:55.465 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-18T06:22:55.465 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2023-01-18T06:22:55.493 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:22:55.521 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:22:55.521 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:22:55.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:22:55.583 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:22:56.295 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:24:56.349 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:24:56.403 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:24:57.105 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:24:57.105 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:24:57.105 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:24:57.105 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:24:57.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:24:57.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:24:57.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:24:57.752 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:24:57.781 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:24:57.781 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:24:57.794 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:24:57.845 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:24:58.517 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:26:58.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:26:58.622 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:26:59.408 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:26:59.409 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:26:59.409 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:26:59.409 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:26:59.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:26:59.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:26:59.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:27:00.090 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:27:00.119 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:27:00.119 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:27:00.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:27:00.186 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:27:00.953 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:29:01.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:29:01.067 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:29:01.852 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:29:01.852 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:29:01.852 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:29:01.852 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:29:01.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:29:01.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:29:02.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:29:02.461 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:29:02.489 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:29:02.490 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:29:02.503 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:29:02.564 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:29:03.297 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:31:03.371 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:31:03.428 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:31:04.224 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:31:04.768 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:31:04.769 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:31:04.769 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:31:04.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:31:04.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:31:05.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:31:05.404 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:31:05.433 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:31:05.433 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:31:05.446 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:31:05.500 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:31:06.262 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:33:06.347 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:33:06.404 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:33:07.187 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:33:07.187 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:33:07.187 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:33:07.187 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:33:07.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:33:07.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:33:07.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:33:07.862 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:33:07.891 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:33:07.892 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:33:07.905 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:33:07.959 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:33:08.703 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:35:08.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-18T06:35:08.815 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:35:09.602 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1'] 2023-01-18T06:35:09.603 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3'] 2023-01-18T06:35:09.603 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.0'] 2023-01-18T06:35:09.603 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:35:09.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2023-01-18T06:35:09.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2023-01-18T06:35:09.964 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-01-18T06:35:10.264 INFO:teuthology.misc:Compressing logs... 2023-01-18T06:35:10.291 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists; not overwritten 2023-01-18T06:35:10.291 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists; not overwritten 2023-01-18T06:35:10.306 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi145 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-18T06:35:10.360 INFO:teuthology.kill:Killing Pids: {31129} 2023-01-18T06:35:11.134 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-18T06:37:11.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...