2023-01-14T00:54:49.525 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2023-01-14T00:54:49.693 INFO:teuthology.lock.ops:Start node 'smithi181.front.sepia.ceph.com' reimaging
2023-01-14T00:54:49.694 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T00:54:49.695 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2023-01-14T00:54:49.712 INFO:teuthology.lock.ops:Node 'smithi181.front.sepia.ceph.com' reimaging is complete
2023-01-14T00:54:49.712 INFO:teuthology.lock.ops:Start node 'smithi191.front.sepia.ceph.com' reimaging
2023-01-14T00:54:49.712 INFO:teuthology.lock.ops:Updating [smithi191.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T00:54:49.713 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server
2023-01-14T00:54:49.741 INFO:teuthology.lock.ops:Node 'smithi191.front.sepia.ceph.com' reimaging is complete
2023-01-14T00:54:49.864 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of rhel 8.4
2023-01-14T00:54:49.911 INFO:teuthology.provision.fog.smithi191:Scheduling deploy of rhel 8.4
2023-01-14T00:54:50.096 INFO:teuthology.orchestra.console:Power off smithi181
2023-01-14T00:54:50.163 INFO:teuthology.orchestra.console:Power off smithi191
2023-01-14T00:55:02.621 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2023-01-14T00:55:02.656 INFO:teuthology.orchestra.console:Power off for smithi191 completed
2023-01-14T00:55:02.723 INFO:teuthology.orchestra.console:Power on smithi181
2023-01-14T00:55:02.758 INFO:teuthology.orchestra.console:Power on smithi191
2023-01-14T00:55:06.988 INFO:teuthology.orchestra.console:Power on for smithi181 completed
2023-01-14T00:55:07.022 INFO:teuthology.orchestra.console:Power on for smithi191 completed
2023-01-14T00:55:07.090 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish
2023-01-14T00:55:07.124 INFO:teuthology.provision.fog.smithi191:Waiting for deploy to finish
2023-01-14T00:58:11.532 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-01-14T00:58:11.534 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.191
2023-01-14T00:58:20.620 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-01-14T00:58:20.621 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.191
2023-01-14T00:58:29.680 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-01-14T00:58:29.681 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.191
2023-01-14T00:58:38.764 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-01-14T00:58:38.765 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.191
2023-01-14T00:58:47.852 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.191
2023-01-14T00:59:44.820 WARNING:teuthology.provision.fog:timed out
2023-01-14T00:59:51.210 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T00:59:51.496 INFO:teuthology.provision.fog.smithi181:Node is ready
2023-01-14T00:59:51.509 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com
2023-01-14T00:59:51.562 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181
2023-01-14T00:59:51.878 INFO:teuthology.provision.fog.smithi181:Deploy complete!
2023-01-14T00:59:53.859 WARNING:teuthology.provision.fog:timed out
2023-01-14T01:00:00.229 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T01:00:00.499 INFO:teuthology.provision.fog.smithi191:Node is ready
2023-01-14T01:00:00.515 INFO:teuthology.orchestra.run.smithi191.stdout:smithi191.front.sepia.ceph.com
2023-01-14T01:00:00.570 INFO:teuthology.orchestra.run.smithi191.stdout:172.21.15.191 smithi191.front.sepia.ceph.com smithi191
2023-01-14T01:00:00.909 INFO:teuthology.provision.fog.smithi191:Deploy complete!
2023-01-14T01:00:01.071 INFO:teuthology.lock.ops:Checking smithi181.front.sepia.ceph.com
2023-01-14T01:00:01.072 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-14T01:00:01.088 INFO:teuthology.lock.ops:Checking smithi191.front.sepia.ceph.com
2023-01-14T01:00:01.089 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-14T01:00:01.105 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: set os type and version on server
2023-01-14T01:00:01.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T01:00:01.302 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64
2023-01-14T01:00:01.358 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Red Hat Enterprise Linux"
2023-01-14T01:00:01.359 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8.4 (Ootpa)"
2023-01-14T01:00:01.359 INFO:teuthology.orchestra.run.smithi181.stdout:ID="rhel"
2023-01-14T01:00:01.359 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="fedora"
2023-01-14T01:00:01.359 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8.4"
2023-01-14T01:00:01.360 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8"
2023-01-14T01:00:01.360 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-01-14T01:00:01.360 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31"
2023-01-14T01:00:01.360 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-01-14T01:00:01.361 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.redhat.com/"
2023-01-14T01:00:01.361 INFO:teuthology.orchestra.run.smithi181.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-01-14T01:00:01.361 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-14T01:00:01.361 INFO:teuthology.orchestra.run.smithi181.stdout:
2023-01-14T01:00:01.361 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-14T01:00:01.361 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-01-14T01:00:01.362 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-01-14T01:00:01.362 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-01-14T01:00:01.363 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2023-01-14T01:00:01.383 INFO:teuthology.lock.ops:Updating [smithi191.front.sepia.ceph.com]: set os type and version on server
2023-01-14T01:00:01.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T01:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64
2023-01-14T01:00:01.579 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="Red Hat Enterprise Linux"
2023-01-14T01:00:01.579 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="8.4 (Ootpa)"
2023-01-14T01:00:01.579 INFO:teuthology.orchestra.run.smithi191.stdout:ID="rhel"
2023-01-14T01:00:01.579 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="fedora"
2023-01-14T01:00:01.580 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="8.4"
2023-01-14T01:00:01.580 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el8"
2023-01-14T01:00:01.580 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-01-14T01:00:01.580 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31"
2023-01-14T01:00:01.581 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-01-14T01:00:01.581 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://www.redhat.com/"
2023-01-14T01:00:01.581 INFO:teuthology.orchestra.run.smithi191.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-01-14T01:00:01.581 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-14T01:00:01.581 INFO:teuthology.orchestra.run.smithi191.stdout:
2023-01-14T01:00:01.582 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-14T01:00:01.582 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-01-14T01:00:01.582 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-01-14T01:00:01.582 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-01-14T01:00:01.583 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server
2023-01-14T01:00:01.683 INFO:teuthology.dispatcher.supervisor:Running job 7141617
2023-01-14T01:00:01.683 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi/7141617 --name yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi --description fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} -- /home/teuthworker/archive/yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi/7141617/orig.config.yaml
2023-01-14T01:00:01.689 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-01-13_17:02:58-fs-wip-yuri5-testing-2023-01-12-1540-quincy-distro-default-smithi/7141617
2023-01-14T01:00:01.689 INFO:teuthology.dispatcher.supervisor:Job PID: 24455
2023-01-14T01:00:01.689 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-01-14T13:01:18.660 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:01:18.819 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:01:19.557 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:01:19.558 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:01:19.559 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:01:19.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:01:19.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:01:20.109 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:01:20.262 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:01:21.333 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-mds.a.log: file size changed while zipping
2023-01-14T13:01:24.177 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-mds.c.log: file size changed while zipping
2023-01-14T13:01:38.844 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping
2023-01-14T13:01:39.242 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping
2023-01-14T13:01:42.772 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping
2023-01-14T13:01:55.039 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping
2023-01-14T13:02:02.687 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping
2023-01-14T13:02:03.312 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping
2023-01-14T13:02:23.135 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping
2023-01-14T13:02:25.975 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping
2023-01-14T13:02:39.642 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping
2023-01-14T13:02:43.081 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping
2023-01-14T13:02:47.763 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping
2023-01-14T13:02:48.941 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping
2023-01-14T13:03:04.160 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping
2023-01-14T13:03:43.720 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:03:44.484 INFO:teuthology.kill:Nuking machines: ['smithi181', 'smithi191']
2023-01-14T13:03:45.080 INFO:teuthology.kill:2023-01-14 13:03:45,080.080 INFO:teuthology.nuke:targets:
2023-01-14T13:03:45.081 INFO:teuthology.kill:  smithi181.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxeA/MzTUUTaJjelYHGtvIUsOpli1pTplwYkwAqfEJR81SFdooeuoNOoo49Kd9A8svkKZOU2F1GUW3klsVjzFfvwHeOeVDa0pQeyxtWO6FQDgekDuGHIVd67kEpfyLqgiuV13UNjh5cMAya6n0aelctQLt0yBf6sel4+OaJ06IkU1A5D9e07Gfmy4uZ9PSq5ZvCetCyCPwLVakYT5i2V/cxSso9QwEEZnExZAuWzFZLHKAyQMbI9wdePO/HjIBaSZUm64O2MELGw/3qcPXg8RC/0ymowM5ml08AXLbYDlVX0BXNDKGOXTWMFY55eeHfDU965HIvIDpvd10E5pRLyS77c0qZfrIiVRUPJfWI7iH8zvx0MhQxhjR3sWAqyn8yH8fQzIYcV7fLwp8va2CPdwKCL0juhYWE5zDIS78SXMwQGnmqeG8ZEkVj6EfGbDRyqo0aGksGJBG1OwWlrzb6VyHGCC3aTh9ZgFsCD6mV/GFu0nX30gAdQa3IWg3mW2CpKU=
2023-01-14T13:03:45.082 INFO:teuthology.kill:  smithi191.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuQPCWbo7QOBD/W4+25NJcHH97Ep7DI0OaP8vkeOyKY1Le5m4cCWWZzGGyLqLHQAqz+QkYhh6tEZp99dm0Xa9aN5QrZn4gBcm5cRCEN7AuycQskxS8TIdyG1cj/ts2txtVSJEPSVYP1MhEbRyU5suZqEnQVhkLZEnvz2VhkTDVO/LikUhLvi/9+12LJU035CLSpOUBr8xpXY5huFxB+cU0iDyuWD+EnjYGSvrs5XL3Y89mDzZM09zqmEYHUmcUKi0WcS/MDJasj/YNafXUow06tRqHXaXGXjyctdjnExgAnUiycsaBZdfCXRTr6IKuGpPDfLmAiMME1hOQGzw6rpgUtu7XINznVpj4QYHBTK0D69+Qx+wFZgtDp0YsI5CHM2+eFfvpiP+fJK7UfoHcs5L9tRoWuG5bTT6btF01LlWEBQ4bzsfzt4XwZ9j4Oa3Nc/HVyW9QGHXyDotbzPTz6Doc/h9NxE4ayuhdzyZZOKu/NNFNUITIP9VSsf2uS4nxwpk=
2023-01-14T13:03:45.082 INFO:teuthology.kill:2023-01-14 13:03:45,080.080 INFO:teuthology.nuke:Checking targets against current locks
2023-01-14T13:03:45.145 INFO:teuthology.kill:2023-01-14 13:03:45,145.145 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-14T13:03:45.174 INFO:teuthology.kill:2023-01-14 13:03:45,173.173 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-14T13:03:45.234 INFO:teuthology.kill:2023-01-14 13:03:45,234.234 INFO:teuthology.orchestra.console:Power off smithi181
2023-01-14T13:03:45.268 INFO:teuthology.kill:2023-01-14 13:03:45,267.267 INFO:teuthology.orchestra.console:Power off smithi191
2023-01-14T13:03:57.743 INFO:teuthology.kill:2023-01-14 13:03:57,743.743 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2023-01-14T13:03:57.749 INFO:teuthology.kill:2023-01-14 13:03:57,749.749 INFO:teuthology.orchestra.console:Power off for smithi191 completed
2023-01-14T13:03:57.887 INFO:teuthology.kill:2023-01-14 13:03:57,886.886 INFO:teuthology.lock.ops:unlocked: smithi181.front.sepia.ceph.com
2023-01-14T13:03:57.889 INFO:teuthology.kill:2023-01-14 13:03:57,889.889 INFO:teuthology.lock.ops:unlocked: smithi191.front.sepia.ceph.com
2023-01-14T13:05:58.060 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:05:58.111 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:05:58.837 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:05:58.838 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:05:58.838 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:05:58.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:05:58.918 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:00.035 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:01.115 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:02.231 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:03.309 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:04.425 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:05.502 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:06.617 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:07.694 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:08.810 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T13:06:08.811 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T13:06:08.867 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:06:09.562 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:08:09.622 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:08:09.680 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:08:10.441 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:08:10.442 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:08:10.442 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:08:10.442 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:08:42.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:08:42.372 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:08:42.992 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:08:43.029 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:08:43.029 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:08:43.066 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:08:43.067 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:08:43.068 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:08:43.098 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:08:43.135 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:08:43.166 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:08:43.166 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:08:43.204 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:08:43.204 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:08:43.205 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:08:43.234 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:08:43.271 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:08:43.301 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:08:43.302 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:08:43.339 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:08:43.339 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:08:43.396 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:08:44.094 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:10:44.151 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:10:44.209 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:10:44.964 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:10:44.965 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:10:44.965 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:10:44.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:10:45.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:10:45.169 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:10:45.339 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:10:45.366 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:10:45.366 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:10:45.406 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:10:45.406 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:10:45.407 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:10:45.434 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:10:45.474 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:10:45.501 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:10:45.502 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:10:45.543 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:10:45.543 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:10:45.545 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:10:45.570 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:10:45.612 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:10:45.640 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:10:45.640 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:10:45.681 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:10:45.682 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:10:45.733 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:10:46.435 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:12:46.496 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:12:46.554 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:12:47.314 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:12:47.315 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:12:47.315 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:12:47.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:12:47.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:12:47.519 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:12:47.700 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:12:47.726 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:12:47.726 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:12:47.769 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:12:47.769 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:12:47.770 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:12:47.925 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:12:47.951 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:12:47.981 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:12:48.018 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:12:48.018 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:12:48.050 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:12:48.051 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:12:48.144 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:12:48.847 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:14:48.898 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:14:48.954 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:14:49.738 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:14:49.739 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:14:49.739 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:14:49.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:14:49.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:14:49.951 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:14:50.130 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:14:50.158 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:14:50.158 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:14:50.198 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:14:50.198 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:14:50.199 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:14:50.351 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:14:50.379 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:14:50.406 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:14:50.448 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:14:50.448 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:14:50.474 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:14:50.474 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:14:50.526 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:14:51.225 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:16:51.280 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:16:51.337 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:16:52.084 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:16:52.085 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:16:52.085 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:16:52.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:16:52.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:16:52.285 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:16:52.425 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:16:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:16:52.455 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:16:52.493 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:16:52.494 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:16:52.495 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:16:52.525 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:16:52.538 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:16:52.593 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:16:53.275 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:18:53.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:18:53.412 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:18:54.140 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:18:54.141 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:18:54.141 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:18:54.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:18:54.323 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:18:54.593 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:18:54.620 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:18:54.635 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:18:54.687 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:18:55.386 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:20:55.439 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:20:55.492 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:20:56.168 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:20:56.169 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:20:56.169 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:20:56.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:20:56.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:20:56.630 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:20:56.661 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:20:56.671 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:20:56.724 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:20:57.389 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:22:57.452 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:22:57.511 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:22:58.245 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:22:58.246 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:22:58.246 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:22:58.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:22:58.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:22:58.721 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:22:58.749 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:22:58.763 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:22:58.817 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:22:59.517 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:24:59.566 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:24:59.619 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:25:00.309 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:25:00.310 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:25:00.310 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:25:00.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:25:00.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:25:00.778 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:25:00.807 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:25:00.821 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:25:00.874 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:25:01.546 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:27:01.597 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:27:01.656 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:27:02.465 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:27:02.466 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:27:02.466 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:27:02.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:27:02.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:27:02.925 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:27:02.954 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:27:02.968 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:27:03.020 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:27:03.729 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:29:03.783 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:29:03.838 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:29:04.636 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:29:04.637 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:29:04.637 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:29:04.637 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:29:04.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:29:05.062 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:29:05.090 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:29:05.104 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:29:05.157 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:29:05.819 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:31:05.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:31:05.943 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:31:06.682 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:31:06.683 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:31:06.683 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:31:06.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:31:25.004 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T13:31:25.073 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:31:25.833 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:33:25.894 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:33:25.950 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:33:26.726 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:33:26.727 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:33:26.727 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:33:26.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:33:29.804 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T13:33:29.861 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:33:30.596 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:35:30.648 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:35:30.703 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:35:31.420 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:35:31.420 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:35:31.421 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:35:31.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:35:34.476 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T13:35:34.530 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:35:35.222 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:37:35.275 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:37:35.338 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:37:36.100 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:37:36.101 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:37:36.101 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:37:36.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:37:36.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:37:36.309 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:37:36.482 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:37:36.540 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:37:36.540 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:37:36.598 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:37:36.599 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:37:36.600 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:37:36.626 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:37:36.629 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:37:36.692 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:37:36.693 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:37:36.720 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:37:36.721 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:37:36.722 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:37:36.759 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:37:36.787 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:37:36.827 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:37:36.827 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:37:36.853 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:37:36.854 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:37:36.912 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:37:37.635 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:39:37.687 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:39:37.747 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:39:38.556 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:39:38.556 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:39:38.557 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:39:38.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:39:38.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:39:38.762 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:39:38.940 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:39:38.969 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:39:38.969 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:39:39.008 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:39:39.009 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:39:39.010 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:39:39.161 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:39:39.187 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:39:39.226 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:39:39.255 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:39:39.256 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:39:39.294 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:39:39.295 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:39:39.353 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:39:40.079 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:41:40.139 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:41:40.199 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:41:40.978 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:41:40.979 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:41:40.980 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:41:40.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:41:41.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:41:41.188 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:41:41.364 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:41:41.392 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:41:41.393 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:41:41.432 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:41:41.432 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:41:41.433 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:41:41.585 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:41:41.612 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:41:41.652 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:41:41.679 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:41:41.679 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:41:41.719 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:41:41.720 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:41:41.776 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:41:42.514 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:43:42.569 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:43:42.621 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:43:43.396 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:43:43.397 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:43:43.397 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:43:43.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:43:43.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:43:43.617 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:43:43.809 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:43:43.841 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:43:43.841 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:43:43.879 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:43:43.880 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:43:43.881 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:43:43.911 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:43:43.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:43:44.004 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:43:44.726 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:45:44.785 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:45:44.875 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:45:45.599 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:45:45.599 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:45:45.599 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:45:45.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:45:45.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:45:46.014 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:45:46.043 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:45:46.056 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:45:46.110 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:45:46.867 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:47:46.926 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:47:46.979 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:47:47.702 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:47:47.703 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:47:47.703 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:47:47.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:47:47.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:47:48.125 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:47:48.153 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:47:48.167 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:47:48.218 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:47:48.960 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:49:49.014 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:49:49.073 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:49:49.885 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:49:49.885 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:49:49.886 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:49:49.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:49:50.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:49:50.314 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:49:50.345 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:49:50.356 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:49:50.412 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:49:51.152 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:51:51.203 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:51:51.276 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:51:52.048 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:51:52.048 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:51:52.049 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:51:52.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:51:52.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:51:52.500 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:51:52.529 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:51:52.542 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:51:52.598 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:51:53.459 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:53:53.509 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:53:53.568 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:53:54.351 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:53:54.351 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:53:54.352 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:53:54.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:53:54.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:53:54.839 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:53:54.868 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:53:54.881 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:53:54.939 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:53:55.671 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:55:55.736 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:55:55.799 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:55:56.560 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:55:56.561 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:55:56.561 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:55:56.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:55:56.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:55:56.988 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:55:57.029 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:55:57.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:55:57.087 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:55:57.837 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:57:57.896 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:57:57.980 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:57:58.786 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:57:58.787 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T13:57:58.787 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:57:58.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T13:57:58.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T13:57:59.244 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:57:59.280 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:57:59.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:57:59.345 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T13:58:00.157 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:00:00.214 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:00:00.273 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:00:00.979 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:00:00.980 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:00:00.980 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:00:00.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:00:01.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:00:01.456 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:00:01.488 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:00:01.498 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:00:01.554 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:00:02.277 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:02:02.329 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:02:02.427 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:02:03.165 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:02:03.166 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:02:03.166 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:02:03.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:02:03.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:02:03.656 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:02:03.684 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:02:03.698 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:02:03.757 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:02:04.525 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:04:04.591 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:04:04.644 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:04:05.383 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:04:05.384 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:04:05.384 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:04:05.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:04:05.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:04:05.814 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:04:05.846 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:04:05.857 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:04:05.910 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:04:06.610 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:06:06.669 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:06:06.728 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:06:07.477 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:06:07.478 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:06:07.478 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:06:07.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:06:07.655 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:06:07.935 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:06:07.963 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:06:07.978 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:06:08.034 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:06:08.742 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:08:08.794 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:08:08.861 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:08:09.707 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:08:09.708 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:08:09.709 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:08:09.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:08:09.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:08:10.140 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:08:10.168 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:08:10.183 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:08:10.254 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:08:11.033 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:10:11.093 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:10:11.152 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:10:11.903 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:10:11.903 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:10:11.904 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:10:11.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:10:30.220 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:10:30.275 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:10:30.977 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:12:31.040 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:12:31.099 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:12:31.887 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:12:31.887 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:12:31.888 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:12:31.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:12:42.064 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:12:42.120 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:12:42.842 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:14:42.894 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:14:42.950 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:14:43.664 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:14:43.665 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:14:43.665 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:14:43.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:14:43.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:14:43.866 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:14:43.996 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:14:44.053 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:14:44.053 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:14:44.246 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:14:44.246 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:14:44.247 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:14:44.275 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T14:14:44.275 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T14:14:44.343 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T14:14:44.343 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:14:44.369 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T14:14:44.369 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:14:44.371 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:14:44.409 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T14:14:44.437 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T14:14:44.476 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T14:14:44.476 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:14:44.504 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T14:14:44.504 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:14:44.557 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:14:45.294 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:16:45.343 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:16:45.403 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:16:46.180 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:16:46.180 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:16:46.181 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:16:46.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:16:46.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:16:46.390 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:16:46.570 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:16:46.600 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:16:46.600 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:16:46.639 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:16:46.639 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:16:46.640 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:16:46.796 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:16:46.822 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T14:16:46.850 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T14:16:46.890 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T14:16:46.890 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:16:46.918 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T14:16:46.918 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:16:47.011 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:16:47.730 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:18:47.781 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:18:47.837 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:18:48.562 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:18:48.563 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:18:48.563 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:18:48.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:18:48.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:18:48.773 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:18:48.953 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:18:48.981 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:18:48.981 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:18:49.021 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:18:49.022 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:18:49.023 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:18:49.174 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:18:49.202 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T14:18:49.240 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T14:18:49.270 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T14:18:49.270 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:18:49.309 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T14:18:49.309 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:18:49.363 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:18:50.033 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:20:50.083 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:20:50.137 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:20:50.888 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:20:50.888 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:20:50.889 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:20:50.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:20:51.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:20:51.094 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:20:51.273 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:20:51.301 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:20:51.301 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:20:51.341 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:20:51.341 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:20:51.342 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:20:51.370 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:20:51.385 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:20:51.440 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:20:52.172 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:22:52.237 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:22:52.294 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:22:53.035 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:22:53.036 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:22:53.036 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:22:53.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:22:53.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:22:53.501 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:22:53.531 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:22:53.544 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:22:53.599 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:22:54.294 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:24:54.357 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:24:54.412 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:24:55.140 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:24:55.141 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:24:55.141 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:24:55.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:24:55.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:24:55.630 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:24:55.661 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:24:55.670 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:24:55.726 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:24:56.420 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:26:56.475 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:26:56.530 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:26:57.260 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:26:57.261 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:26:57.261 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:26:57.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:26:57.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:26:57.752 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:26:57.785 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:26:57.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:26:57.840 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:26:58.562 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:28:58.613 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:28:58.669 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:28:59.402 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:28:59.403 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:28:59.403 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:28:59.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:28:59.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:28:59.874 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:28:59.902 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:28:59.907 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:28:59.962 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:29:00.655 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:31:00.708 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:31:00.767 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:31:01.504 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:31:01.504 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:31:01.505 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:31:01.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:31:01.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:31:01.986 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:31:02.015 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:31:02.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:31:02.074 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:31:02.795 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:33:02.853 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:33:02.907 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:33:03.625 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:33:03.626 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:33:03.626 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:33:03.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:33:03.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:33:04.092 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:33:04.128 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:33:04.130 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:33:04.217 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:33:04.929 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:35:04.989 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:35:05.043 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:35:05.912 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:35:05.913 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:35:05.913 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:35:05.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:35:06.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:35:06.398 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:35:06.427 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:35:06.430 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:35:06.520 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:35:07.267 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:37:07.323 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:37:07.379 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:37:08.073 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:37:08.074 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:37:08.074 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:37:08.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:37:08.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:37:08.550 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:37:08.582 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:37:08.584 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:37:08.637 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:37:09.354 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:39:09.441 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:39:09.498 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:39:10.233 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:39:10.233 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:39:10.233 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:39:10.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:39:10.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:39:10.709 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:39:10.740 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:39:10.744 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:39:10.799 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:39:11.514 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:41:11.565 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:41:11.620 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:41:12.380 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:41:12.381 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:41:12.381 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:41:12.381 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:41:12.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:41:12.766 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:41:12.795 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:41:12.801 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:41:12.854 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:41:13.540 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:43:13.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:43:13.658 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:43:14.473 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:43:14.474 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:43:14.474 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:43:14.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:43:14.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:43:14.981 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:43:15.012 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:43:15.014 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:43:15.070 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:43:15.817 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:45:15.869 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:45:15.931 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:45:16.699 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:45:16.700 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:45:16.700 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:45:16.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:45:16.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T14:45:17.151 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:45:17.182 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:45:17.184 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:45:17.248 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:45:17.939 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:47:17.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:47:18.052 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:47:18.936 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:47:18.936 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:47:18.937 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:47:18.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:47:53.548 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:47:53.606 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:47:54.417 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:49:54.471 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:49:54.583 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:49:55.386 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:49:55.386 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:49:55.387 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:49:55.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:49:58.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:49:58.502 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:49:59.212 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:51:59.268 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:51:59.325 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:52:00.058 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:52:00.058 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:52:00.059 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:52:00.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:52:03.120 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:52:03.177 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:52:03.918 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:54:03.982 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:54:04.043 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:54:04.814 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:54:04.815 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:54:04.815 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:54:04.816 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:54:07.884 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:54:07.940 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:54:08.677 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:56:08.733 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:56:08.791 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:56:09.560 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:56:09.561 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:56:09.561 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:56:09.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:56:12.620 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:56:12.679 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:56:13.423 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:58:13.482 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:58:13.537 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:58:14.255 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:58:14.256 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T14:58:14.256 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:58:14.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T14:58:17.324 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T14:58:17.379 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T14:58:18.098 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:00:18.153 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:00:18.223 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:00:19.057 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:00:19.058 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:00:19.058 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:00:19.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:00:19.180 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:20.298 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:21.376 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:22.496 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:23.613 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:24.692 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:25.810 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:26.889 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:28.007 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:29.125 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:29.126 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T15:00:29.185 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:00:29.881 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:02:29.931 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:02:29.985 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:02:30.762 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:02:30.763 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:02:30.763 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:02:30.763 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:02:33.836 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T15:02:33.890 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:02:34.568 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:04:34.623 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:04:34.682 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:04:35.434 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:04:35.435 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:04:35.435 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:04:35.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:04:35.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:04:35.626 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:04:35.844 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:04:35.844 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:35.909 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:04:35.926 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-14T15:04:36.001 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T15:04:36.002 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:36.071 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:04:36.087 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:04:36.439 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping
2023-01-14T15:04:36.750 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping
2023-01-14T15:04:37.047 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping
2023-01-14T15:04:37.079 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:04:37.079 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:37.392 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:04:38.102 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:06:38.162 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:06:38.268 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:06:39.049 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:06:39.049 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:06:39.049 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:06:39.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:06:49.424 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T15:06:49.481 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:06:50.271 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:08:50.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:08:50.410 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:08:51.212 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:08:51.213 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:08:51.213 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:08:51.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:08:51.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:08:51.420 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:08:51.630 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:08:51.630 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:08:51.691 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:08:51.844 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:08:51.862 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:08:51.914 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:08:51.915 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:08:52.426 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:08:53.171 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:10:53.223 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:10:53.284 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:10:54.114 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:10:54.115 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:10:54.115 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:10:54.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:10:54.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:10:54.308 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:11:12.848 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T15:11:12.909 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:11:13.654 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:13:13.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:13:13.772 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:13:14.556 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:13:14.557 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:13:14.557 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:13:14.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:13:14.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:13:17.804 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T15:13:17.862 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:13:18.657 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:15:18.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:15:18.781 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:15:19.643 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:15:19.644 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:15:19.644 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:15:19.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:15:19.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:15:22.860 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T15:15:22.916 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:15:23.645 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:17:23.698 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:17:23.756 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:17:24.542 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:17:24.543 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:17:24.543 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:17:24.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:17:24.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:17:27.788 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T15:17:27.846 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:17:28.549 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:19:28.610 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:19:28.667 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:19:29.394 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:19:29.394 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:19:29.395 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:19:29.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:20:01.043 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:02.154 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:03.266 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:04.378 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:05.489 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:06.600 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:07.711 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:08.821 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:09.930 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:11.001 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:20:11.002 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T15:20:11.055 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:20:11.716 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:22:11.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:22:11.837 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:22:12.651 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:22:12.652 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:22:12.652 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:22:12.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:22:22.796 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T15:22:22.899 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:22:23.645 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:24:23.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:24:23.757 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:24:24.510 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:24:24.511 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:24:24.511 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:24:24.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:24:24.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:24:24.719 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:24:24.903 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:24:24.941 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:24:24.942 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:24:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:24:25.023 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:24:25.025 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:24:25.181 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:24:25.210 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T15:24:25.232 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T15:24:25.281 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:24:25.281 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:24:25.308 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:24:25.309 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:24:25.367 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:24:26.184 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:26:26.242 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:26:26.302 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:26:27.101 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:26:27.102 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:26:27.102 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:26:27.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:26:27.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:26:27.308 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:26:27.494 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:26:27.524 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:26:27.524 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:26:27.563 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:26:27.563 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:26:27.565 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:26:27.633 INFO:teuthology.orchestra.run.smithi191.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:26:27.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:26:27.690 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:26:28.436 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:28:28.493 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:28:28.567 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:28:29.371 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:28:29.372 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:28:29.372 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:28:29.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:28:29.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:28:29.537 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:28:29.668 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:28:29.697 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:28:29.697 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:28:29.735 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:28:29.736 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:28:29.737 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:28:29.765 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:28:29.779 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:28:29.839 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:28:30.592 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:30:30.643 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:30:30.698 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:30:31.377 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:30:31.378 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:30:31.378 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:30:31.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:30:31.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:30:31.580 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:30:31.750 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:30:31.750 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:30:31.793 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:30:31.823 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:30:31.835 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:30:31.888 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:30:32.562 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:32:32.610 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:32:32.669 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:32:33.374 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:32:33.375 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:32:33.375 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:32:33.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:32:33.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:32:33.843 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:32:33.872 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:32:33.885 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:32:33.941 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:32:34.645 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:34:34.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:34:34.755 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:34:35.486 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:34:35.487 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:34:35.487 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:34:35.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:34:35.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:34:35.925 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:34:35.954 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:34:35.964 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:34:36.016 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:34:36.704 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:36:36.755 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:36:36.821 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:36:37.564 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:36:37.564 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:36:37.565 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:36:37.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:36:37.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:36:38.057 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:36:38.088 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:36:38.090 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:36:38.148 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:36:38.901 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:38:38.975 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:38:39.033 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:38:39.877 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:38:39.878 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:38:39.878 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:38:39.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:38:40.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:38:40.365 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:38:40.393 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:38:40.399 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:38:40.457 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:38:41.206 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:40:41.265 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:40:41.325 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:40:42.101 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:40:42.102 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:40:42.102 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:40:42.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:40:42.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:40:42.570 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:40:42.599 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:40:42.603 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:40:42.662 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:40:43.451 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:42:45.900 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:42:45.964 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:42:46.762 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:42:46.763 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:42:46.763 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:42:46.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:42:46.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:43:05.424 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T15:43:05.488 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:43:06.289 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:45:18.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:45:18.240 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:45:19.011 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:45:19.011 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:45:19.012 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:45:19.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:45:19.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:45:22.220 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T15:45:22.276 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:45:23.006 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:47:23.110 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:47:23.164 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:47:23.959 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:47:23.960 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:47:23.960 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:47:23.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:47:42.412 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T15:47:42.464 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:47:43.205 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:49:43.263 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:49:43.327 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:49:44.095 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:49:44.095 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:49:44.096 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:49:44.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:49:44.172 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:45.247 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:46.366 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:47.481 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:48.557 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:49.671 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:50.746 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:51.859 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:52.936 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:54.012 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:49:54.012 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T15:49:54.064 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:49:54.802 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:51:54.861 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:51:54.925 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:51:55.652 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:51:55.653 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:51:55.653 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:51:55.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:52:11.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:52:11.697 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:52:11.795 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:52:11.844 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:52:11.844 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:52:11.861 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:52:11.862 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:52:11.864 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:52:11.901 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-14T15:52:11.911 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T15:52:11.978 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T15:52:11.978 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:52:11.995 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T15:52:11.995 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:52:11.997 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:52:12.045 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T15:52:12.053 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:52:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:52:12.112 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:52:12.127 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:52:12.127 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:52:12.189 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:52:12.977 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:54:13.033 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:54:13.091 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:54:13.865 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:54:13.866 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:54:13.866 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:54:13.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:54:14.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:54:14.069 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:54:17.100 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T15:54:17.156 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:54:17.841 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:56:17.897 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:56:18.009 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:56:18.804 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:56:18.804 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:56:18.805 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:56:18.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:56:18.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:56:19.011 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:56:19.182 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:56:19.219 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:56:19.219 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:56:19.263 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:56:19.263 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:56:19.265 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:56:19.420 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:56:19.437 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:56:19.448 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T15:56:19.516 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:56:19.516 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:56:19.533 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:56:19.534 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:56:19.591 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:56:29.155 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:58:29.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:58:29.268 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:58:30.076 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:58:30.077 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T15:58:30.077 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:58:30.078 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T15:58:30.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T15:58:30.282 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T15:58:30.435 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:58:30.463 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:58:30.463 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:58:30.479 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:58:30.480 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:58:30.481 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:58:30.532 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T15:58:30.533 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T15:58:30.590 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T15:58:31.287 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:00:31.341 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:00:31.399 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:00:32.163 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:00:32.164 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:00:32.164 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:00:32.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:00:32.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:00:32.479 INFO:teuthology.orchestra.run.smithi181.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T16:00:32.637 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:00:32.638 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:00:32.658 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:00:32.801 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T16:00:32.806 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T16:00:32.863 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:00:33.574 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:02:33.638 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:02:33.696 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:02:34.481 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:02:34.482 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:02:34.483 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:02:34.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:02:34.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:02:34.944 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:02:34.992 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T16:02:34.994 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T16:02:35.058 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:02:35.842 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:04:35.909 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:04:35.965 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:04:36.718 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:04:36.719 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:04:36.719 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:04:36.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:04:36.855 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:04:37.190 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:04:37.216 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-14T16:04:37.226 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T16:04:37.228 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi181 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T16:04:37.283 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:04:37.983 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:06:38.033 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:06:38.094 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:06:38.914 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:06:38.915 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:06:38.915 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:06:38.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:06:39.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:06:57.488 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi191
2023-01-14T16:06:57.541 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:06:58.255 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:08:58.312 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:08:58.367 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:08:59.128 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:08:59.129 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:08:59.129 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:08:59.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:09:17.516 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T16:09:17.570 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:09:18.323 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:11:18.376 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:11:18.464 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:11:19.214 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:11:19.214 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:11:19.215 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:11:19.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:11:19.292 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:20.407 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:21.520 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:22.596 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:23.672 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:24.788 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:25.901 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:27.014 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:28.088 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:29.201 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-01-14T16:11:29.201 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T16:11:29.284 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:11:30.005 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:13:30.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:13:30.112 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:13:30.848 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:13:30.849 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:13:30.849 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:13:30.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:13:31.568 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi181
2023-01-14T16:13:31.623 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:13:32.334 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:15:32.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:15:32.454 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:15:33.273 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:15:33.274 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:15:33.274 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:15:33.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:15:33.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:15:33.499 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T16:15:33.683 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T16:15:33.703 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:15:33.704 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:15:33.755 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:15:33.756 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:15:33.758 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:15:33.773 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-14T16:15:33.795 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T16:15:33.796 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:15:33.827 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:15:33.841 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T16:15:33.878 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T16:15:33.901 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T16:15:33.901 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:15:33.916 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T16:15:33.916 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:15:33.976 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:15:34.699 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:17:34.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:17:34.813 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:17:35.590 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:17:35.590 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:17:35.591 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:17:35.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:17:37.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:17:37.934 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T16:17:38.059 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T16:17:38.133 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:17:38.134 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:17:38.151 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:17:38.151 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:17:38.153 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:17:38.169 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-14T16:17:38.184 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T16:17:38.185 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:17:38.214 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:17:38.230 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T16:17:38.269 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T16:17:38.293 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T16:17:38.294 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:17:38.310 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T16:17:38.310 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:17:38.369 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:17:39.064 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:19:39.117 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:19:39.174 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:19:39.969 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:19:39.970 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:19:39.970 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:19:39.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:19:40.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:19:40.163 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T16:19:40.333 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T16:19:40.352 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:19:40.352 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:19:40.369 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:19:40.370 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:19:40.372 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:19:40.500 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:19:40.517 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T16:19:40.519 INFO:teuthology.orchestra.run.smithi191.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T16:19:40.535 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T16:19:40.535 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:19:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T16:19:40.579 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:19:40.638 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:19:41.390 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:21:41.446 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:21:41.504 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:21:42.327 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:21:42.328 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:21:42.328 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:21:42.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:21:42.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:21:42.523 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T16:21:42.881 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T16:21:42.881 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:21:42.902 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:21:43.031 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:21:43.047 INFO:teuthology.orchestra.run.smithi181.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T16:21:43.066 INFO:teuthology.orchestra.run.smithi181.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T16:21:43.067 INFO:teuthology.orchestra.run.smithi181.stderr:tar: Error is not recoverable: exiting now
2023-01-14T16:21:43.142 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:21:43.871 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T16:23:43.944 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T16:23:44.001 INFO:teuthology.kill:Killing Pids: {24455}
2023-01-14T16:23:44.777 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T16:23:44.777 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-01-14T16:23:44.778 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:23:44.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-01-14T16:23:44.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com'
2023-01-14T16:23:45.233 INFO:teuthology.misc:Compressing logs...
2023-01-14T16:23:45.345 INFO:teuthology.misc:Compressing logs...