2023-01-14T10:52:45.958 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-14T10:52:46.142 INFO:teuthology.lock.ops:Start node 'smithi016.front.sepia.ceph.com' reimaging 2023-01-14T10:52:46.143 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: reset os type and version on server 2023-01-14T10:52:46.144 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2023-01-14T10:52:46.160 INFO:teuthology.lock.ops:Node 'smithi016.front.sepia.ceph.com' reimaging is complete 2023-01-14T10:52:46.160 INFO:teuthology.lock.ops:Start node 'smithi033.front.sepia.ceph.com' reimaging 2023-01-14T10:52:46.161 INFO:teuthology.lock.ops:Updating [smithi033.front.sepia.ceph.com]: reset os type and version on server 2023-01-14T10:52:46.161 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-01-14T10:52:46.177 INFO:teuthology.lock.ops:Node 'smithi033.front.sepia.ceph.com' reimaging is complete 2023-01-14T10:52:46.309 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of ubuntu 20.04 2023-01-14T10:52:46.338 INFO:teuthology.provision.fog.smithi033:Scheduling deploy of ubuntu 20.04 2023-01-14T10:52:46.510 INFO:teuthology.orchestra.console:Power off smithi016 2023-01-14T10:52:46.550 INFO:teuthology.orchestra.console:Power off smithi033 2023-01-14T10:52:46.659 INFO:teuthology.orchestra.console:Power off for smithi016 completed 2023-01-14T10:52:46.690 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2023-01-14T10:52:46.761 INFO:teuthology.orchestra.console:Power on smithi016 2023-01-14T10:52:46.791 INFO:teuthology.orchestra.console:Power on smithi033 2023-01-14T10:52:51.017 INFO:teuthology.orchestra.console:Power on for smithi016 completed 2023-01-14T10:52:51.040 INFO:teuthology.orchestra.console:Power on for smithi033 completed 2023-01-14T10:52:51.118 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish 2023-01-14T10:52:51.141 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish 2023-01-14T10:56:40.656 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-01-14T10:56:40.658 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16 2023-01-14T10:56:49.740 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-01-14T10:56:49.741 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16 2023-01-14T10:56:58.828 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.33 2023-01-14T10:56:58.829 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.16 2023-01-14T10:58:04.892 WARNING:teuthology.provision.fog:timed out 2023-01-14T10:58:04.894 WARNING:teuthology.provision.fog:timed out 2023-01-14T10:58:11.113 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-14T10:58:11.133 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-01-14T10:58:11.647 INFO:teuthology.provision.fog.smithi033:Node is ready 2023-01-14T10:58:11.654 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033.front.sepia.ceph.com 2023-01-14T10:58:11.661 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33 smithi033.front.sepia.ceph.com smithi033 2023-01-14T10:58:11.674 INFO:teuthology.provision.fog.smithi016:Node is ready 2023-01-14T10:58:11.680 INFO:teuthology.orchestra.run.smithi016.stdout:smithi016.front.sepia.ceph.com 2023-01-14T10:58:11.729 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16 smithi016.front.sepia.ceph.com smithi016 2023-01-14T10:58:11.848 INFO:teuthology.provision.fog.smithi033:Deploy complete! 2023-01-14T10:58:11.957 INFO:teuthology.provision.fog.smithi016:Deploy complete! 2023-01-14T10:58:12.274 INFO:teuthology.lock.ops:Checking smithi016.front.sepia.ceph.com 2023-01-14T10:58:12.275 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-14T10:58:12.293 INFO:teuthology.lock.ops:Checking smithi033.front.sepia.ceph.com 2023-01-14T10:58:12.293 INFO:teuthology.lock.ops:New key found. Updating... 2023-01-14T10:58:12.310 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: set os type and version on server 2023-01-14T10:58:12.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T10:58:12.487 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2023-01-14T10:58:12.494 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Ubuntu" 2023-01-14T10:58:12.494 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-14T10:58:12.494 INFO:teuthology.orchestra.run.smithi016.stdout:ID=ubuntu 2023-01-14T10:58:12.494 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE=debian 2023-01-14T10:58:12.494 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-14T10:58:12.495 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="20.04" 2023-01-14T10:58:12.495 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-14T10:58:12.495 INFO:teuthology.orchestra.run.smithi016.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-14T10:58:12.495 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-14T10:58:12.495 INFO:teuthology.orchestra.run.smithi016.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-14T10:58:12.496 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_CODENAME=focal 2023-01-14T10:58:12.496 INFO:teuthology.orchestra.run.smithi016.stdout:UBUNTU_CODENAME=focal 2023-01-14T10:58:12.497 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2023-01-14T10:58:12.515 INFO:teuthology.lock.ops:Updating [smithi033.front.sepia.ceph.com]: set os type and version on server 2023-01-14T10:58:12.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T10:58:12.687 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2023-01-14T10:58:12.739 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="Ubuntu" 2023-01-14T10:58:12.739 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-14T10:58:12.739 INFO:teuthology.orchestra.run.smithi033.stdout:ID=ubuntu 2023-01-14T10:58:12.740 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE=debian 2023-01-14T10:58:12.740 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-14T10:58:12.740 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="20.04" 2023-01-14T10:58:12.740 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-14T10:58:12.740 INFO:teuthology.orchestra.run.smithi033.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-14T10:58:12.741 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-14T10:58:12.741 INFO:teuthology.orchestra.run.smithi033.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-14T10:58:12.741 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_CODENAME=focal 2023-01-14T10:58:12.741 INFO:teuthology.orchestra.run.smithi033.stdout:UBUNTU_CODENAME=focal 2023-01-14T10:58:12.741 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2023-01-14T10:58:12.845 INFO:teuthology.dispatcher.supervisor:Running job 7141968 2023-01-14T10:58:12.845 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-13_17:05:26-rbd-wip-yuri3-testing-2023-01-12-1538-quincy-distro-default-smithi/7141968 --name yuriw-2023-01-13_17:05:26-rbd-wip-yuri3-testing-2023-01-12-1538-quincy-distro-default-smithi --description rbd/thrash/{base/install clusters/{fixed-2 openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} thrashers/cache thrashosds-health workloads/rbd_api_tests_journaling} -- /home/teuthworker/archive/yuriw-2023-01-13_17:05:26-rbd-wip-yuri3-testing-2023-01-12-1538-quincy-distro-default-smithi/7141968/orig.config.yaml 2023-01-14T10:58:12.849 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-01-13_17:05:26-rbd-wip-yuri3-testing-2023-01-12-1538-quincy-distro-default-smithi/7141968 2023-01-14T10:58:12.850 INFO:teuthology.dispatcher.supervisor:Job PID: 27981 2023-01-14T10:58:12.850 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-01-14T22:58:17.873 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T22:58:18.025 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T22:58:18.821 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T22:58:18.822 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T22:58:18.823 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:58:18.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T22:58:19.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T22:58:19.256 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:58:19.424 INFO:teuthology.misc:Compressing logs... 2023-01-14T22:58:29.760 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-01-14T22:58:40.490 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-14T22:58:44.674 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2023-01-14T22:58:49.189 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-01-14T22:58:49.532 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-01-14T22:59:10.195 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2023-01-14T22:59:17.888 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-14T22:59:34.055 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-14T22:59:37.797 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-01-14T23:00:07.586 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-01-14T23:00:30.296 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-14T23:12:56.342 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:12:57.085 INFO:teuthology.kill:Nuking machines: ['smithi016', 'smithi033'] 2023-01-14T23:12:57.660 INFO:teuthology.kill:2023-01-14 23:12:57,660.660 INFO:teuthology.nuke:targets: 2023-01-14T23:12:57.661 INFO:teuthology.kill: smithi016.front.sepia.ceph.com: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw 2023-01-14T23:12:57.662 INFO:teuthology.kill: smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= 2023-01-14T23:12:57.662 INFO:teuthology.kill:2023-01-14 23:12:57,660.660 INFO:teuthology.nuke:Checking targets against current locks 2023-01-14T23:12:57.711 INFO:teuthology.kill:2023-01-14 23:12:57,711.711 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-14T23:12:57.736 INFO:teuthology.kill:2023-01-14 23:12:57,736.736 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-14T23:12:57.789 INFO:teuthology.kill:2023-01-14 23:12:57,788.788 INFO:teuthology.orchestra.console:Power off smithi016 2023-01-14T23:12:57.829 INFO:teuthology.kill:2023-01-14 23:12:57,829.829 INFO:teuthology.orchestra.console:Power off smithi033 2023-01-14T23:13:10.282 INFO:teuthology.kill:2023-01-14 23:13:10,282.282 INFO:teuthology.orchestra.console:Power off for smithi016 completed 2023-01-14T23:13:10.306 INFO:teuthology.kill:2023-01-14 23:13:10,305.305 INFO:teuthology.orchestra.console:Power off for smithi033 completed 2023-01-14T23:13:10.425 INFO:teuthology.kill:2023-01-14 23:13:10,425.425 INFO:teuthology.lock.ops:unlocked: smithi016.front.sepia.ceph.com 2023-01-14T23:13:10.445 INFO:teuthology.kill:2023-01-14 23:13:10,445.445 INFO:teuthology.lock.ops:unlocked: smithi033.front.sepia.ceph.com 2023-01-14T23:15:10.655 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:15:10.702 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:15:11.468 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:15:11.468 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:15:11.469 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:15:11.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:15:29.868 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:15:29.911 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:15:30.642 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:17:30.793 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:17:30.834 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:17:31.572 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:17:31.573 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:17:31.573 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:17:31.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:17:31.652 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:32.729 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:33.845 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:34.959 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:36.073 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:37.187 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:38.302 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:39.415 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:40.529 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:41.642 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-14T23:17:41.643 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:17:41.684 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:17:42.402 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:19:42.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:19:42.498 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:19:43.177 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:19:43.178 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:19:43.178 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:19:43.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:19:43.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:19:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:19:44.805 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:19:44.930 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:19:44.931 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:19:44.958 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:19:44.958 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:19:44.960 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:19:44.977 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T23:19:45.020 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-14T23:19:45.038 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T23:19:45.038 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:19:45.053 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-14T23:19:45.053 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:19:45.055 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:19:45.070 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:19:45.072 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:19:45.137 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:19:45.137 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:19:45.155 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:19:45.156 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:19:45.197 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:19:45.854 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:21:45.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:21:45.953 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:21:46.725 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:21:46.726 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:21:46.726 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:21:46.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:22:46.729 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:22:46.789 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:22:47.524 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:24:47.577 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:24:47.624 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:24:48.434 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:24:48.435 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:24:48.435 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:24:48.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:24:48.604 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:24:48.631 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:24:48.795 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:24:48.874 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:24:48.875 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:24:48.898 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:24:48.898 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:24:48.900 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:24:48.999 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:24:49.015 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:24:49.016 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:24:49.071 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:24:49.072 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:24:49.087 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:24:49.088 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:24:49.133 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:24:49.843 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:26:49.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:26:49.942 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:26:50.681 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:26:50.682 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:26:50.682 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:26:50.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:26:50.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:26:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:26:51.068 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:26:51.085 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:26:51.086 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:26:51.100 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:26:51.101 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:26:51.102 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:26:51.224 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:26:51.246 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:26:51.250 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:26:51.259 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:26:51.260 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:26:51.309 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:26:51.309 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:26:51.385 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:26:52.068 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:28:52.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:28:52.160 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:28:52.864 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:28:52.865 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:28:52.865 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:28:52.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:28:53.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:28:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:28:53.246 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-14T23:28:53.265 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:28:53.265 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:28:53.279 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-14T23:28:53.280 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:28:53.281 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:28:53.396 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:28:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:28:53.412 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-14T23:28:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:28:53.470 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:28:53.484 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-14T23:28:53.485 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-14T23:28:53.529 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:28:54.189 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:30:54.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:30:54.283 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:30:55.000 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:30:55.001 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:30:55.001 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:30:55.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:30:55.174 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:30:55.393 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:30:55.514 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:30:55.655 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:30:56.355 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:32:56.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:32:56.456 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:32:57.206 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:32:57.206 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:32:57.207 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:32:57.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:32:57.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:32:57.672 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:32:57.803 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:32:58.312 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-01-14T23:33:02.362 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log: file size changed while zipping 2023-01-14T23:33:02.415 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-14T23:33:02.608 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-14T23:33:17.552 INFO:teuthology.orchestra.run.smithi016.stderr:tar: .: file changed as we read it 2023-01-14T23:33:20.080 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:33:20.801 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:35:20.851 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:35:20.894 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:35:21.630 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:35:21.631 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:35:21.631 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:35:21.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:35:21.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:35:22.056 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:35:22.139 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:35:22.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:35:22.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:35:22.184 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:35:22.225 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:35:22.901 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:37:22.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:37:23.013 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:37:23.715 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:37:23.716 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:37:23.716 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:37:23.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:37:23.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:37:24.142 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:37:24.243 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:37:24.262 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:37:24.263 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:37:24.265 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:37:24.265 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:37:24.265 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:37:24.306 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:37:24.977 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:39:25.026 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:39:25.078 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:39:25.861 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:39:25.862 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:39:25.862 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:39:25.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:39:26.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:39:26.318 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:39:26.443 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:39:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:39:26.463 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:39:26.464 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:39:26.464 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:39:26.465 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:39:26.516 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:39:27.261 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:41:27.310 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:41:27.354 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:41:28.131 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:41:28.132 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:41:28.132 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:41:28.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:41:28.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-14T23:41:28.544 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:41:28.670 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:41:28.689 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:41:28.690 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:41:28.692 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-14T23:41:28.696 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-14T23:41:28.700 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-14T23:41:28.742 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:41:29.467 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:43:29.514 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:43:29.557 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:43:30.304 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:43:30.305 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:43:30.305 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:43:30.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:44:30.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:44:30.351 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:44:31.048 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:46:31.097 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:46:31.139 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:46:31.904 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:46:31.905 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:46:31.905 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:46:31.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:46:34.988 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:46:35.031 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:46:35.748 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:48:35.811 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:48:35.857 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:48:36.593 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:48:36.594 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:48:36.594 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:48:36.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:48:39.660 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:48:39.704 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:48:40.395 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:50:40.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:50:40.498 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:50:41.264 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:50:41.265 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:50:41.265 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:50:41.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:50:44.336 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:50:44.380 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:50:45.121 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:52:45.183 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:52:45.225 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:52:45.951 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:52:45.951 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:52:45.952 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:52:45.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:52:49.004 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:52:49.048 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:52:49.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:54:49.825 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:54:49.869 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:54:50.644 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:54:50.645 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:54:50.645 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:54:50.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:54:53.711 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:54:53.755 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:54:54.505 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:56:54.554 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:56:54.597 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:56:55.334 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:56:55.335 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:56:55.335 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:56:55.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:56:58.412 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:56:58.451 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:56:59.153 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-14T23:58:59.202 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-14T23:58:59.245 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:58:59.988 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-14T23:58:59.989 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-14T23:58:59.989 INFO:teuthology.misc:Compressing logs... 2023-01-14T23:58:59.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-14T23:59:03.052 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-14T23:59:03.095 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-14T23:59:03.870 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:01:03.917 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:01:03.958 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:01:04.691 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:01:04.692 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:01:04.692 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:01:04.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:01:07.756 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:01:07.797 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:01:08.471 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:03:08.527 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:03:08.567 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:03:09.337 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:03:09.540 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:03:09.540 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:03:09.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:03:12.620 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:03:12.672 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:03:13.403 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:05:13.452 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:05:13.493 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:05:14.157 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:05:14.158 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:05:14.158 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:05:14.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:05:17.228 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:05:17.271 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:05:17.958 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:07:18.008 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:07:18.052 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:07:18.811 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:07:18.811 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:07:18.811 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:07:18.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:07:21.868 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:07:21.910 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:07:22.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:09:22.642 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:09:22.685 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:09:23.386 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:09:23.387 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:09:23.387 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:09:23.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:09:26.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:09:26.484 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:09:27.224 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:11:27.274 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:11:27.315 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:11:28.086 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:11:28.087 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:11:28.087 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:11:28.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:11:31.148 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:11:31.219 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:11:31.917 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:13:31.965 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:13:32.013 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:13:32.722 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:13:32.723 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:13:32.723 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:13:32.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:13:35.788 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:13:35.834 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:13:36.566 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:15:36.632 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:15:36.670 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:15:37.429 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:15:37.429 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:15:37.430 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:15:37.430 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:15:40.492 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:15:40.534 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:15:41.178 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:17:41.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:17:41.269 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:17:42.094 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:17:42.095 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:17:42.095 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:17:42.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:17:45.164 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:17:45.206 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:17:45.898 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:19:45.947 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:19:45.989 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:19:46.709 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:19:46.709 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:19:46.710 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:19:46.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:19:49.772 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:19:49.815 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:19:50.590 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:21:50.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:21:50.681 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:21:51.433 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:21:51.433 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:21:51.433 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:21:51.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:21:54.508 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:21:54.550 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:21:55.239 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:23:55.289 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:23:55.335 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:23:56.079 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:23:56.080 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:23:56.081 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:23:56.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:23:59.148 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:23:59.195 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:23:59.886 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:25:59.939 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:25:59.981 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:26:00.699 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:26:00.700 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:26:00.700 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:26:00.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:26:03.760 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:26:03.800 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:26:04.480 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:28:04.532 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:28:04.574 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:28:05.327 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:28:05.328 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:28:05.328 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:28:05.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:28:08.396 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:28:08.438 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:28:09.147 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:30:09.195 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:30:09.237 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:30:09.985 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:30:09.986 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:30:09.986 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:30:09.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:30:13.068 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:30:13.112 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:30:13.792 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:32:13.847 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:32:13.889 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:32:14.641 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:32:14.642 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:32:14.642 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:32:14.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:32:17.708 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:32:17.749 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:32:18.431 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:34:18.483 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:34:18.524 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:34:19.225 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:34:19.225 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:34:19.226 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:34:19.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:34:22.284 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:34:22.324 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:34:23.027 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:36:23.074 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:36:23.114 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:36:23.804 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:36:23.805 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:36:23.805 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:36:23.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:36:26.860 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:36:26.930 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:36:27.587 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:38:27.633 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:38:27.672 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:38:28.436 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:38:28.436 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:38:28.437 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:38:28.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:38:31.500 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:38:31.542 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:38:32.230 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:40:32.281 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:40:32.324 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:40:33.044 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:40:33.045 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:40:33.045 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:40:33.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:40:36.108 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:40:36.150 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:40:36.824 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:42:36.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:42:36.922 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:42:37.695 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:42:37.696 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:42:37.696 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:42:37.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:42:56.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:42:56.119 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:42:56.825 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:44:56.899 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:44:56.941 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:44:57.668 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:44:57.669 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:44:57.669 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:44:57.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:44:57.745 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:44:58.858 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:44:59.933 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:45:01.048 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:45:02.162 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:45:03.235 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:45:04.309 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:45:05.493 ERROR:teuthology.orchestra.connection:Error authenticating with smithi016.front.sepia.ceph.com: Authentication failed. 2023-01-15T00:46:06.496 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi016 2023-01-15T00:46:06.542 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:46:07.248 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:48:07.310 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:48:07.357 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:48:08.102 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:48:08.103 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:48:08.103 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:48:08.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:48:08.276 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T00:48:08.287 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T00:48:08.511 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T00:48:08.512 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:48:08.584 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:48:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-15T00:48:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-15T00:48:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:48:08.916 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:48:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T00:48:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-15T00:48:08.961 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:48:09.059 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:48:09.809 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:50:09.856 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:50:09.896 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:50:10.625 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:50:10.626 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:50:10.626 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:50:10.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:50:11.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T00:50:11.656 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T00:50:11.836 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T00:50:11.836 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:50:11.876 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:50:11.893 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-15T00:50:11.962 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-15T00:50:11.963 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:50:12.034 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T00:50:12.209 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-01-15T00:50:12.296 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-01-15T00:50:12.419 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-01-15T00:50:12.441 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-01-15T00:50:12.566 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-01-15T00:50:12.663 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-15T00:50:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2023-01-15T00:50:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:50:17.990 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:50:18.654 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:52:18.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:52:18.760 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:52:19.496 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:52:19.497 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:52:19.497 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:52:19.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:52:19.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T00:52:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T00:52:19.930 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T00:52:19.931 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:52:19.987 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:52:20.008 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-01-15T00:52:22.596 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:52:22.611 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T00:52:22.631 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:52:22.632 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:52:22.633 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:52:22.673 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:52:23.342 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:54:23.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:54:23.449 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:54:24.143 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:54:24.144 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:54:24.144 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:54:24.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:54:24.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T00:54:24.337 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T00:54:24.544 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T00:54:24.545 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2023-01-15T00:54:24.600 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:54:24.960 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:54:24.978 INFO:teuthology.orchestra.run.smithi016.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T00:54:24.994 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:54:24.994 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:54:24.995 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:54:25.241 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:54:25.898 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:56:25.947 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:56:25.988 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:56:26.673 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:56:26.674 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:56:26.674 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:56:26.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:56:26.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T00:56:27.138 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:56:27.500 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:56:27.534 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:56:27.534 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:56:27.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:56:27.577 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:56:28.233 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T00:58:28.292 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T00:58:28.335 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:58:29.047 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T00:58:29.047 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T00:58:29.048 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:58:29.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T00:58:29.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T00:58:29.785 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:58:30.068 INFO:teuthology.misc:Compressing logs... 2023-01-15T00:58:30.141 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-01-15T00:58:30.210 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T00:58:30.210 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T00:58:30.213 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi033 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T00:58:30.253 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T00:58:30.972 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:00:31.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:00:31.081 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:00:31.780 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:00:31.781 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:00:31.781 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:00:31.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:00:31.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:00:32.686 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:00:33.095 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:00:33.200 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:00:33.228 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:00:33.241 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:00:33.280 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:00:33.926 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:02:33.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:02:34.017 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:02:34.794 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:02:34.795 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:02:34.795 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:02:34.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:02:34.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:02:35.999 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:02:36.468 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:02:36.493 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:02:36.494 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:02:36.541 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:02:36.545 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:02:36.549 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:02:36.619 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:02:37.313 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:04:37.364 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:04:37.405 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:04:38.123 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:04:38.123 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:04:38.124 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:04:38.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:04:38.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:04:38.615 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:04:39.348 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:04:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:04:39.372 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:04:39.375 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:04:39.386 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:04:39.389 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:04:39.390 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:04:39.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:04:39.436 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:04:40.106 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:06:40.156 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:06:40.201 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:06:41.086 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:06:41.086 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:06:41.087 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:06:41.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:06:41.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:06:41.557 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:06:41.735 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:06:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:06:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:06:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:06:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:06:41.755 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:06:41.796 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:06:42.536 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:08:42.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:08:42.632 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:08:43.350 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:08:43.351 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:08:43.351 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:08:43.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:08:43.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:08:43.861 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:08:44.023 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:08:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:08:44.043 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:08:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:08:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:08:44.048 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:08:44.090 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:08:44.777 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:10:44.828 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:10:44.869 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:10:45.573 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:10:45.573 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:10:45.574 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:10:45.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:10:45.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:10:46.031 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:10:46.996 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:10:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:10:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:10:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:10:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:10:47.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:10:47.060 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:10:47.749 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:12:47.804 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:12:47.844 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:12:48.574 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:12:48.575 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:12:48.575 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:12:48.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:12:48.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:12:49.034 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:12:49.392 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:12:49.413 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:12:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:12:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:12:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:12:49.415 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:12:49.458 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:12:50.146 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:14:50.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:14:50.241 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:14:50.973 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:14:50.973 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:14:50.973 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:14:50.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:14:51.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:14:51.426 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:14:51.780 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:14:51.800 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:14:51.801 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:14:51.801 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:14:51.801 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:14:51.802 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:14:51.849 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:14:52.519 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:16:52.582 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:16:52.623 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:16:53.378 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:16:53.378 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:16:53.379 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:16:53.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:16:53.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:17:12.012 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T01:17:12.054 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:17:12.803 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:19:12.850 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:19:12.894 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:19:13.629 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:19:13.630 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:19:13.630 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:19:13.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:19:13.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:19:13.889 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:15.006 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:16.124 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:17.203 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:18.323 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:19.443 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:20.563 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:21.681 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:22.760 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:23.877 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T01:19:23.878 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T01:19:23.923 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:19:24.616 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:21:24.670 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:21:24.717 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:21:25.483 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:21:25.483 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:21:25.484 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:21:25.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:21:25.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:22:25.662 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T01:22:25.706 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:22:26.448 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:24:26.500 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:24:26.542 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:24:27.241 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:24:27.242 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:24:27.242 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:24:27.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:24:27.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:25:27.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T01:25:27.462 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:25:28.187 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:27:28.241 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:27:28.323 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:27:29.102 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:27:29.103 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:27:29.103 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:27:29.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:27:29.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:27:29.482 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T01:27:29.559 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T01:27:29.560 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T01:27:29.561 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:27:29.856 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:27:29.876 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T01:27:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:27:29.877 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:27:29.877 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:27:29.877 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:27:29.879 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:27:29.922 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:27:30.708 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:29:30.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:29:30.803 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:29:31.523 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:29:31.523 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:29:31.524 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:29:31.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:29:31.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:29:31.881 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T01:29:31.919 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T01:29:31.920 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T01:29:31.922 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:29:32.252 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:29:32.272 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T01:29:32.273 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:29:32.273 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:29:32.273 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:29:32.274 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:29:32.275 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:29:32.315 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:29:33.029 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:31:33.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:31:33.126 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:31:33.843 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:31:33.844 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:31:33.844 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:31:33.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:31:34.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:31:34.208 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T01:31:34.244 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T01:31:34.245 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T01:31:34.247 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:31:34.376 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:31:34.392 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T01:31:34.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:31:34.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:31:34.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:31:34.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:31:34.397 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:31:34.437 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:31:35.119 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:33:35.172 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:33:35.215 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:33:35.956 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:33:35.957 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:33:35.957 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:33:35.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:33:36.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:33:36.317 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T01:33:36.354 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T01:33:36.354 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T01:33:36.355 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:33:37.095 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:33:37.113 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T01:33:37.116 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:33:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:33:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:33:37.117 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:33:37.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:33:37.160 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:33:37.816 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:35:37.871 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:35:37.915 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:35:38.671 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:35:38.671 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:35:38.672 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:35:38.672 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:35:38.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:35:39.021 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T01:35:39.081 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T01:35:39.082 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T01:35:39.083 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:35:40.388 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:35:40.408 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T01:35:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:35:40.410 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:35:40.410 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:35:40.410 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:35:40.411 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:35:40.456 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:35:41.178 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:37:41.229 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:37:41.270 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:37:42.025 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:37:42.025 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:37:42.025 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:37:42.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:37:42.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:37:42.439 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:37:43.592 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:37:43.613 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:37:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:37:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:37:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:37:43.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:37:43.657 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:37:44.326 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:39:44.377 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:39:44.421 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:39:45.119 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:39:45.119 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:39:45.324 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:39:45.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:39:45.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:39:45.704 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:39:49.908 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:39:49.929 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:39:49.929 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:39:49.930 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:39:49.930 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:39:49.931 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:39:50.172 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:39:50.840 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:41:50.891 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:41:50.931 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:41:51.626 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:41:51.626 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:41:51.627 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:41:51.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:41:51.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:41:52.013 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:41:56.212 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:41:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:41:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:41:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:41:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:41:56.236 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:41:56.274 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:41:56.939 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:43:57.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:43:57.239 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:43:57.983 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:43:57.984 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:43:57.984 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:43:57.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:43:58.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:43:58.386 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:44:02.592 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:44:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:44:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:44:02.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:44:02.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:44:02.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:44:02.657 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:44:03.333 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:46:03.399 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:46:03.442 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:46:04.190 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:46:05.208 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:46:05.208 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:46:05.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:46:05.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:46:05.658 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:46:05.782 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:46:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:46:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:46:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:46:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:46:05.803 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:46:05.846 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:46:06.503 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:48:06.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:48:06.598 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:48:07.324 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:48:07.325 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:48:07.325 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:48:07.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:48:07.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:48:07.761 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:48:10.944 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:48:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:48:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:48:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:48:10.965 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:48:10.966 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:48:11.012 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:48:11.689 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:50:11.754 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:50:11.795 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:50:12.542 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:50:12.542 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:50:12.542 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:50:12.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:50:12.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:50:12.931 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:50:16.104 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:50:16.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:50:16.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:50:16.127 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:50:16.127 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:50:16.127 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:50:16.128 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:50:16.170 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:50:16.845 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:52:16.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:52:16.940 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:52:17.619 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:52:17.620 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:52:17.620 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:52:17.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:52:17.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:52:18.041 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:52:21.592 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:52:21.612 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:52:21.612 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T01:52:21.612 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:52:21.613 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T01:52:21.613 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:52:21.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:52:21.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:52:21.614 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:52:21.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:52:21.654 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:52:22.324 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:54:22.380 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:54:22.626 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:54:23.343 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:54:23.344 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:54:23.344 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:54:23.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:54:23.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:54:23.796 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:54:25.147 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:54:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:54:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:54:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:54:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:54:25.190 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:54:25.233 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:54:26.037 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:56:26.087 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:56:26.127 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:56:26.833 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:56:26.836 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:56:26.836 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:56:26.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:56:27.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:56:27.241 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:56:29.964 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:56:29.986 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:56:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:56:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:56:30.188 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:56:30.191 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:56:30.191 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T01:56:30.191 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:56:30.191 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T01:56:30.192 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T01:56:30.192 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:56:30.235 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:56:30.963 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T01:58:31.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T01:58:31.060 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:58:31.811 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T01:58:31.811 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T01:58:31.812 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:58:31.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T01:58:31.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T01:58:32.214 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:58:35.396 INFO:teuthology.misc:Compressing logs... 2023-01-15T01:58:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T01:58:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T01:58:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:58:35.620 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:58:35.622 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T01:58:35.622 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T01:58:35.622 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T01:58:35.623 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T01:58:35.623 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T01:58:35.623 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T01:58:35.664 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T01:58:36.330 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:00:36.385 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:00:36.428 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:00:37.124 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:00:37.125 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:00:37.125 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:00:37.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:00:37.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:00:37.515 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:00:37.624 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:00:37.656 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:00:37.656 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T02:00:37.656 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:00:37.657 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T02:00:37.657 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T02:00:37.658 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:00:37.658 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:00:37.658 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:00:37.658 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:00:37.659 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:00:37.700 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:00:38.392 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:02:38.447 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:02:38.488 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:02:39.271 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:02:39.272 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:02:39.272 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:02:39.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:02:39.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:02:39.671 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:02:41.628 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:02:41.646 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:02:41.646 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T02:02:41.647 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:02:41.647 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T02:02:41.647 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T02:02:41.650 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:02:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:02:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:02:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:02:41.854 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:02:41.896 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:02:42.566 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:04:42.622 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:05:00.251 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:05:01.601 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:05:01.602 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:05:01.602 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:05:01.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:05:01.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:05:01.996 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:05:02.316 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:05:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:05:02.336 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:05:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:05:02.337 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:05:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:05:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T02:05:02.338 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:05:02.339 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T02:05:02.339 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T02:05:02.339 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:05:02.382 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:05:03.085 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:07:03.142 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:07:03.188 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:07:03.863 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:07:03.864 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:07:03.864 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:07:03.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:07:04.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:07:04.281 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:07:18.896 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:07:19.119 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:07:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T02:07:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:07:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T02:07:19.120 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T02:07:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:07:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:07:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:07:19.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:07:19.124 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:07:19.169 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:07:19.915 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:09:19.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:09:20.005 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:09:20.716 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:09:20.717 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:09:20.717 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:09:20.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:09:20.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:09:21.115 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:09:21.848 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:09:21.865 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:09:21.866 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T02:09:21.866 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:09:21.866 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T02:09:21.867 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T02:09:21.867 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:09:21.867 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:09:21.868 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:09:21.868 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:09:21.869 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:09:21.909 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:09:22.566 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:11:22.620 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:11:22.667 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:11:23.399 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:11:23.400 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:11:23.400 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:11:23.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:11:23.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:11:23.831 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:11:23.962 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:11:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:11:23.981 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:11:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:11:23.982 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:11:24.004 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:11:24.045 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:11:24.729 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:13:24.781 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:13:24.822 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:13:25.547 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:13:25.547 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:13:25.548 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:13:25.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:13:25.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:13:25.961 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:13:26.696 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:13:26.716 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:13:26.717 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T02:13:26.717 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:13:26.717 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T02:13:26.717 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T02:13:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:13:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:13:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:13:26.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:13:26.719 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:13:26.758 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:13:27.438 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:15:27.489 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:15:27.529 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:15:28.191 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:15:28.192 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:15:28.192 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:15:28.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:15:28.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:15:28.595 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:15:29.540 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:15:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:15:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:15:29.561 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:15:29.562 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:15:29.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:15:29.605 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:15:30.293 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:17:30.346 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:17:30.387 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:17:31.064 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:17:31.064 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:17:31.065 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:17:31.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:17:31.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:17:31.473 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:17:31.992 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:17:32.013 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:17:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:17:32.216 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:17:32.217 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:17:32.219 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:17:32.220 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.1.log.gz already exists; not overwritten 2023-01-15T02:17:32.220 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:17:32.220 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.0.log.gz already exists; not overwritten 2023-01-15T02:17:32.220 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/ceph-osd.3.log.gz already exists; not overwritten 2023-01-15T02:17:32.221 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:17:32.262 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:17:32.944 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:19:33.002 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:19:33.042 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:19:33.723 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:19:33.724 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:19:33.724 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:19:33.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:19:33.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:20:08.524 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T02:20:08.563 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:20:09.221 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:22:09.273 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:22:09.314 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:22:10.080 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:22:10.080 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:22:10.081 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:22:10.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:22:10.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:22:17.663 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:18.780 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:19.898 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:21.014 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:22.130 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:23.247 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:24.364 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:25.441 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:26.520 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:27.635 ERROR:teuthology.orchestra.connection:Error authenticating with smithi033.front.sepia.ceph.com: Authentication failed. 2023-01-15T02:22:27.636 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T02:22:27.676 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:22:28.408 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:24:28.469 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:24:28.515 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:24:29.272 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:24:29.273 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:24:29.273 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:24:29.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:24:29.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:24:32.524 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T02:24:32.565 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:24:33.271 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:26:33.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:26:33.370 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:26:34.134 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:26:34.134 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:26:34.135 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:26:34.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:26:34.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:26:34.699 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T02:26:34.754 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T02:26:34.754 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T02:26:34.756 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:26:34.793 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory 2023-01-15T02:26:36.270 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2023-01-15T02:26:36.270 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T02:26:36.272 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:26:36.286 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T02:26:36.290 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:26:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:26:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:26:36.291 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:26:36.292 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:26:36.333 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:26:37.004 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:28:37.062 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:28:37.107 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:28:37.907 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:28:37.908 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:28:37.908 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:28:37.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:28:38.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:29:38.095 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi033 2023-01-15T02:29:38.344 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:29:39.007 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:31:39.058 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T02:32:04.430 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:32:05.144 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T02:32:05.145 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T02:32:05.145 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:32:05.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T02:32:05.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T02:32:05.796 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T02:32:05.877 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T02:32:05.878 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T02:32:05.886 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:32:09.360 INFO:teuthology.misc:Compressing logs... 2023-01-15T02:32:09.382 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T02:32:09.382 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T02:32:09.382 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T02:32:09.383 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T02:32:09.383 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T02:32:09.627 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T02:32:10.299 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T02:34:10.351 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T03:04:29.546 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T03:04:30.313 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T03:04:30.313 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T03:04:30.314 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:04:30.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T03:04:30.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T03:04:30.976 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T03:04:31.072 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T03:04:31.072 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T03:04:31.079 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:05:04.764 INFO:teuthology.misc:Compressing logs... 2023-01-15T03:05:04.986 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T03:31:16.364 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T03:31:16.572 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T03:31:17.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T03:31:17.394 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/log/ceph': No such file or directory 2023-01-15T03:31:17.394 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T03:31:17.645 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T03:31:18.391 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T03:33:20.692 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T05:06:09.480 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T05:06:10.313 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T05:06:10.516 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T05:06:10.516 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:06:10.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T05:06:10.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T05:06:11.143 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T05:06:33.345 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T05:06:33.345 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T05:06:33.353 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:06:34.323 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:06:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T05:06:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T05:06:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T05:06:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T05:06:34.378 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T05:06:34.421 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T05:06:35.194 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T05:09:31.068 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-01-15T05:09:33.821 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T05:09:34.947 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T05:09:34.948 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T05:09:34.948 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:09:34.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2023-01-15T05:09:35.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2023-01-15T05:09:35.597 INFO:teuthology.orchestra.run.smithi033.stderr:find: '/var/lib/ceph/crash': No such file or directory 2023-01-15T05:09:35.695 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory 2023-01-15T05:09:35.695 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2023-01-15T05:09:35.705 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:09:35.908 INFO:teuthology.misc:Compressing logs... 2023-01-15T05:09:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-01-15T05:09:35.928 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-01-15T05:09:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.audit.log.gz already exists; not overwritten 2023-01-15T05:09:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/ceph.log.gz already exists; not overwritten 2023-01-15T05:09:35.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi016 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-01-15T05:09:35.971 INFO:teuthology.kill:Killing Pids: {27981} 2023-01-15T05:09:36.640 INFO:teuthology.kill:No locked machines. Not nuking anything 2023-01-15T05:11:36.693 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...