2023-01-14T00:29:53.683 INFO:root:teuthology version: 0.0.1.dev18+g19d18db
2023-01-14T00:29:53.800 INFO:teuthology.lock.ops:Start node 'smithi052.front.sepia.ceph.com' reimaging
2023-01-14T00:29:53.801 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T00:29:53.801 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server
2023-01-14T00:29:53.817 INFO:teuthology.lock.ops:Node 'smithi052.front.sepia.ceph.com' reimaging is complete
2023-01-14T00:29:53.817 INFO:teuthology.lock.ops:Start node 'smithi057.front.sepia.ceph.com' reimaging
2023-01-14T00:29:53.817 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: reset os type and version on server
2023-01-14T00:29:53.818 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2023-01-14T00:29:53.833 INFO:teuthology.lock.ops:Node 'smithi057.front.sepia.ceph.com' reimaging is complete
2023-01-14T00:29:53.962 INFO:teuthology.provision.fog.smithi052:Scheduling deploy of rhel 8.6
2023-01-14T00:29:53.984 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of rhel 8.6
2023-01-14T00:29:54.296 INFO:teuthology.orchestra.console:Power off smithi052
2023-01-14T00:29:54.311 INFO:teuthology.orchestra.console:Power off smithi057
2023-01-14T00:29:54.444 INFO:teuthology.orchestra.console:Power off for smithi052 completed
2023-01-14T00:29:54.455 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2023-01-14T00:29:54.546 INFO:teuthology.orchestra.console:Power on smithi052
2023-01-14T00:29:54.557 INFO:teuthology.orchestra.console:Power on smithi057
2023-01-14T00:29:58.810 INFO:teuthology.orchestra.console:Power on for smithi057 completed
2023-01-14T00:29:58.911 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish
2023-01-14T00:30:02.913 INFO:teuthology.orchestra.console:Power on for smithi052 completed
2023-01-14T00:30:03.014 INFO:teuthology.provision.fog.smithi052:Waiting for deploy to finish
2023-01-14T00:32:36.748 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52
2023-01-14T00:32:45.836 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52
2023-01-14T00:32:54.892 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52
2023-01-14T00:33:03.180 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.57
2023-01-14T00:33:03.952 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52
2023-01-14T00:33:12.236 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.57
2023-01-14T00:33:13.004 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52
2023-01-14T00:33:21.292 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.57
2023-01-14T00:33:30.348 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.57
2023-01-14T00:33:50.786 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T00:33:51.093 INFO:teuthology.provision.fog.smithi052:Node is ready
2023-01-14T00:33:51.120 INFO:teuthology.orchestra.run.smithi052.stdout:smithi022
2023-01-14T00:33:51.178 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-01-14T00:33:51.527 INFO:teuthology.provision.fog.smithi052:Deploy complete!
2023-01-14T00:34:36.393 WARNING:teuthology.provision.fog:timed out
2023-01-14T00:34:42.630 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-14T00:34:42.968 INFO:teuthology.provision.fog.smithi057:Node is ready
2023-01-14T00:34:42.988 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com
2023-01-14T00:34:43.047 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057
2023-01-14T00:34:43.403 INFO:teuthology.provision.fog.smithi057:Deploy complete!
2023-01-14T00:34:43.569 INFO:teuthology.lock.ops:Checking smithi052.front.sepia.ceph.com
2023-01-14T00:34:43.570 INFO:teuthology.lock.ops:Checking smithi057.front.sepia.ceph.com
2023-01-14T00:34:43.572 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: set os type and version on server
2023-01-14T00:34:43.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T00:34:43.833 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64
2023-01-14T00:34:43.891 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Red Hat Enterprise Linux"
2023-01-14T00:34:43.892 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="8.6 (Ootpa)"
2023-01-14T00:34:43.892 INFO:teuthology.orchestra.run.smithi052.stdout:ID="rhel"
2023-01-14T00:34:43.892 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="fedora"
2023-01-14T00:34:43.893 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="8.6"
2023-01-14T00:34:43.893 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el8"
2023-01-14T00:34:43.893 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-01-14T00:34:43.893 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31"
2023-01-14T00:34:43.893 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-01-14T00:34:43.894 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.redhat.com/"
2023-01-14T00:34:43.894 INFO:teuthology.orchestra.run.smithi052.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-01-14T00:34:43.894 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-14T00:34:43.894 INFO:teuthology.orchestra.run.smithi052.stdout:
2023-01-14T00:34:43.894 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-14T00:34:43.895 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-01-14T00:34:43.895 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-01-14T00:34:43.895 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-01-14T00:34:43.896 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server
2023-01-14T00:34:43.920 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: set os type and version on server
2023-01-14T00:34:43.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T00:34:44.110 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64
2023-01-14T00:34:44.168 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Red Hat Enterprise Linux"
2023-01-14T00:34:44.168 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8.6 (Ootpa)"
2023-01-14T00:34:44.168 INFO:teuthology.orchestra.run.smithi057.stdout:ID="rhel"
2023-01-14T00:34:44.169 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="fedora"
2023-01-14T00:34:44.169 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8.6"
2023-01-14T00:34:44.169 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8"
2023-01-14T00:34:44.169 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-01-14T00:34:44.169 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31"
2023-01-14T00:34:44.170 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-01-14T00:34:44.170 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.redhat.com/"
2023-01-14T00:34:44.170 INFO:teuthology.orchestra.run.smithi057.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-01-14T00:34:44.170 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-14T00:34:44.171 INFO:teuthology.orchestra.run.smithi057.stdout:
2023-01-14T00:34:44.171 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-14T00:34:44.171 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-01-14T00:34:44.171 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-01-14T00:34:44.172 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-01-14T00:34:44.172 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2023-01-14T00:34:44.266 INFO:teuthology.dispatcher.supervisor:Running job 7141349
2023-01-14T00:34:44.267 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/virtualenv/bin/teuthology -v --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7141349 --name rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} -- /home/teuthworker/archive/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7141349/orig.config.yaml
2023-01-14T00:34:44.272 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rishabh-2023-01-13_12:08:33-fs-wip-rishabh-testing-2022Nov24-11Jan2023-distro-default-smithi/7141349
2023-01-14T00:34:44.272 INFO:teuthology.dispatcher.supervisor:Job PID: 25351
2023-01-14T00:34:44.273 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-01-14T12:36:27.243 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T12:36:27.419 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:36:28.126 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T12:36:28.127 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T12:36:28.127 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:36:28.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T12:36:28.342 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T12:36:28.796 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:36:29.013 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:36:53.940 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping
2023-01-14T12:37:45.732 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping
2023-01-14T12:38:29.041 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping
2023-01-14T12:39:22.574 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping
2023-01-14T12:39:28.484 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-mds.f.log: file size changed while zipping
2023-01-14T12:40:23.733 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mds.a.log: file size changed while zipping
2023-01-14T12:40:56.026 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping
2023-01-14T12:41:01.321 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping
2023-01-14T12:41:13.691 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping
2023-01-14T12:41:30.611 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mds.e.log: file size changed while zipping
2023-01-14T12:41:34.037 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping
2023-01-14T12:41:44.688 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping
2023-01-14T12:41:50.335 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-mds.b.log: file size changed while zipping
2023-01-14T12:43:20.455 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping
2023-01-14T12:44:17.588 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mds.c.log: file size changed while zipping
2023-01-14T12:45:44.680 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping
2023-01-14T12:48:04.694 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-client.0.104565.log: file size changed while zipping
2023-01-14T12:48:32.915 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping
2023-01-14T12:48:33.649 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping
2023-01-14T12:52:22.849 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:52:23.595 INFO:teuthology.kill:Nuking machines: ['smithi052', 'smithi057']
2023-01-14T12:52:24.188 INFO:teuthology.kill:2023-01-14 12:52:24,188.188 INFO:teuthology.nuke:targets:
2023-01-14T12:52:24.189 INFO:teuthology.kill:  smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-01-14T12:52:24.189 INFO:teuthology.kill:  smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=
2023-01-14T12:52:24.190 INFO:teuthology.kill:2023-01-14 12:52:24,188.188 INFO:teuthology.nuke:Checking targets against current locks
2023-01-14T12:52:24.236 INFO:teuthology.kill:2023-01-14 12:52:24,236.236 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-14T12:52:24.263 INFO:teuthology.kill:2023-01-14 12:52:24,263.263 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-14T12:52:24.312 INFO:teuthology.kill:2023-01-14 12:52:24,312.312 INFO:teuthology.orchestra.console:Power off smithi052
2023-01-14T12:52:24.353 INFO:teuthology.kill:2023-01-14 12:52:24,352.352 INFO:teuthology.orchestra.console:Power off smithi057
2023-01-14T12:52:32.693 INFO:teuthology.kill:2023-01-14 12:52:32,693.693 INFO:teuthology.orchestra.console:Power off for smithi052 completed
2023-01-14T12:52:32.725 INFO:teuthology.kill:2023-01-14 12:52:32,725.725 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2023-01-14T12:52:32.834 INFO:teuthology.kill:2023-01-14 12:52:32,833.833 INFO:teuthology.lock.ops:unlocked: smithi052.front.sepia.ceph.com
2023-01-14T12:52:32.863 INFO:teuthology.kill:2023-01-14 12:52:32,863.863 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com
2023-01-14T12:54:33.077 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T12:54:33.140 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:54:33.915 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T12:54:33.916 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T12:54:33.916 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:54:33.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T12:54:33.997 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:35.112 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:36.229 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:37.303 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:38.418 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:39.532 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:40.647 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:41.761 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:42.876 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:43.990 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T12:54:43.991 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi052
2023-01-14T12:54:44.051 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:54:44.824 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T12:56:44.877 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T12:56:44.945 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:56:45.728 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T12:56:45.728 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T12:56:45.729 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:56:45.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T12:56:46.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T12:56:46.390 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T12:57:02.395 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T12:57:02.441 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T12:57:02.441 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:57:02.516 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T12:57:02.516 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:57:02.518 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:57:02.546 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T12:57:02.547 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T12:57:02.617 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T12:57:02.617 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:57:02.644 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T12:57:02.644 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:57:02.646 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:57:02.686 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T12:57:02.716 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T12:57:02.755 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T12:57:02.756 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:57:02.788 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T12:57:02.788 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:57:02.856 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:57:03.567 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T12:59:03.619 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T12:59:03.683 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:59:04.406 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T12:59:04.407 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T12:59:04.407 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:59:04.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T12:59:04.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T12:59:04.574 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T12:59:04.753 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T12:59:04.784 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T12:59:04.784 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:59:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T12:59:04.823 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:59:04.824 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:59:04.978 INFO:teuthology.misc:Compressing logs...
2023-01-14T12:59:05.005 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T12:59:05.045 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T12:59:05.076 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T12:59:05.076 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:59:05.113 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T12:59:05.113 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T12:59:05.175 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T12:59:05.875 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:01:05.929 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:01:05.988 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:01:06.775 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:01:06.775 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:01:06.776 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:01:06.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:01:06.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:01:06.944 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:01:07.122 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:01:07.150 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:01:07.150 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:01:07.191 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:01:07.191 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:01:07.192 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:01:07.348 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:01:07.376 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:01:07.417 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:01:07.445 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:01:07.446 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:01:07.485 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:01:07.486 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:01:07.549 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:01:08.243 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:03:08.301 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:03:08.364 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:03:09.146 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:03:09.147 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:03:09.147 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:03:09.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:03:09.340 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:03:09.366 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:03:09.553 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:03:09.582 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:03:09.582 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:03:09.621 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:03:09.622 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:03:09.623 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:03:09.652 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:03:09.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:03:09.728 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:03:10.544 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:05:10.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:05:10.658 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:05:11.360 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:05:11.361 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:05:11.361 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:05:11.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:05:11.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:05:11.576 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:05:11.751 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:05:11.782 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:05:11.783 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:05:11.819 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:05:11.819 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:05:11.820 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:05:11.859 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:05:11.863 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:05:11.921 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:05:12.599 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:07:12.650 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:07:12.711 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:07:13.442 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:07:13.443 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:07:13.443 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:07:13.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:07:13.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:07:13.907 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:07:13.935 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:07:13.949 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:07:14.011 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:07:14.695 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:09:14.757 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:09:14.820 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:09:15.546 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:09:15.547 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:09:15.547 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:09:15.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:09:15.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:09:16.031 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:09:16.070 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:09:16.073 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:09:16.137 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:09:16.824 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:11:16.878 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:11:16.943 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:11:17.770 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:11:17.771 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:11:17.771 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:11:17.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:11:17.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:11:18.209 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:11:18.239 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:11:18.251 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:11:18.317 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:11:19.080 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:13:19.136 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:13:19.206 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:13:19.937 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:13:19.937 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:13:19.938 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:13:19.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:13:20.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:13:20.391 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:13:20.425 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:13:20.433 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:13:20.494 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:13:21.223 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:15:21.274 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:15:21.340 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:15:22.092 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:15:22.092 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:15:22.093 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:15:22.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:15:22.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:15:22.579 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:15:22.611 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:15:22.621 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:15:22.718 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:15:23.450 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:17:23.498 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:17:23.554 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:17:24.276 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:17:24.276 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:17:24.277 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:17:24.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:17:24.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:17:24.776 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:17:24.810 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:17:24.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:17:24.872 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:17:25.586 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:19:25.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:19:25.698 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:19:26.539 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:19:26.540 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:19:26.540 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:19:26.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:19:26.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:19:27.031 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:19:27.060 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:19:27.072 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:19:27.129 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:19:27.838 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:21:27.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:21:27.963 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:21:28.673 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:21:28.673 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:21:28.673 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:21:28.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:21:28.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:21:29.218 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:21:29.249 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:21:29.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:21:29.321 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:21:30.029 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:23:30.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:23:30.160 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:23:30.937 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:23:30.938 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:23:30.938 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:23:30.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:23:31.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:23:31.465 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:23:31.497 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:23:31.507 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:23:31.572 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:23:32.292 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:25:32.344 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:25:32.406 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:25:33.172 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:25:33.173 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:25:33.173 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:25:33.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:25:33.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:25:33.749 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:25:33.783 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:25:33.785 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:25:33.847 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:25:34.602 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:27:34.654 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:27:34.721 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:27:35.463 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:27:35.464 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:27:35.464 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:27:35.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:27:35.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:27:35.988 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:27:36.021 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:27:36.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:27:36.101 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:27:36.781 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:29:36.837 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:29:36.894 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:29:37.577 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:29:37.577 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:29:37.578 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:29:37.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:29:37.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:29:38.057 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:29:38.090 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:29:38.100 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:29:38.157 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:29:38.855 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:31:38.906 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:31:39.015 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:31:39.779 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:31:39.780 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:31:39.780 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:31:39.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:31:39.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:31:40.265 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:31:40.299 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:31:40.307 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:31:40.371 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:31:41.084 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:33:41.142 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:33:41.219 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:33:41.937 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:33:41.938 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:33:41.938 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:33:41.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:33:42.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:33:42.416 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:33:42.452 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:33:42.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:33:42.519 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:33:43.261 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:35:43.329 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:35:43.428 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:35:44.157 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:35:44.157 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:35:44.158 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:35:44.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:35:44.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:35:44.646 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:35:44.676 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T13:35:44.677 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:35:44.689 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:35:44.751 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:35:45.426 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:37:45.489 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:37:45.555 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:37:46.305 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:37:46.306 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:37:46.307 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:37:46.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:37:46.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:37:46.840 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:37:46.873 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T13:37:46.874 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:37:46.875 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:37:46.943 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:37:47.690 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:39:47.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:39:47.833 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:39:48.601 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:39:48.602 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:39:48.602 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:39:48.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:39:48.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:39:49.093 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:39:49.126 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T13:39:49.126 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:39:49.135 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:39:49.199 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:39:49.928 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:41:49.988 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:41:50.071 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:41:50.840 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:41:50.840 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:41:50.841 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:41:50.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:41:51.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:41:51.330 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:41:51.361 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T13:41:51.361 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:41:51.372 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:41:51.434 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:41:52.132 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:43:52.191 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:43:52.253 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:43:53.015 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:43:53.016 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:43:53.016 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:43:53.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:43:53.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:43:53.516 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:43:53.549 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T13:43:53.549 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:43:53.558 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:43:53.621 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:43:54.305 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:45:54.359 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:45:54.423 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:45:55.133 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:45:55.134 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:45:55.134 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:45:55.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:45:55.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:45:55.570 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:45:55.599 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T13:45:55.599 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:45:55.613 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:45:55.713 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:45:56.433 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:47:56.504 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:47:56.564 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:47:57.295 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:47:57.295 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:47:57.296 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:47:57.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:47:57.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:47:57.768 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:47:57.797 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T13:47:57.798 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T13:47:57.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T13:47:57.871 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:47:58.570 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:49:58.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:49:58.699 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:49:59.478 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:49:59.478 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:49:59.479 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:49:59.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:50:17.804 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi052
2023-01-14T13:50:17.863 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:50:18.591 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:52:18.646 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:52:18.708 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:52:19.552 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:52:19.552 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:52:19.553 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:52:19.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:52:22.608 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi052
2023-01-14T13:52:22.670 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:52:23.408 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:54:23.457 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:54:23.526 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:54:24.279 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:54:24.279 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:54:24.280 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:54:24.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:54:24.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:54:24.440 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:54:24.578 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:54:24.620 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:54:24.620 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:54:24.672 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:54:24.673 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:54:24.674 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:54:24.701 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:54:24.705 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/home/ubuntu/cephtest/archive’: No such file or directory
2023-01-14T13:54:24.770 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:54:24.771 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:54:24.801 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T13:54:24.801 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:54:24.802 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:54:24.839 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:54:24.871 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:54:24.907 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:54:24.908 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:54:24.939 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:54:24.940 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:54:25.002 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:54:25.745 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:56:25.808 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:56:25.870 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:56:26.623 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:56:26.624 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:56:26.624 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:56:26.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:56:26.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:56:26.835 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:56:26.974 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:56:27.003 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:56:27.004 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:56:27.043 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:56:27.044 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:56:27.045 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:56:27.214 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:56:27.242 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:56:27.280 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:56:27.311 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:56:27.311 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:56:27.349 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:56:27.350 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:56:27.418 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:56:28.147 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T13:58:28.209 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T13:58:28.275 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:58:29.031 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T13:58:29.032 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T13:58:29.032 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:58:29.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T13:58:29.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T13:58:29.237 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:58:29.418 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T13:58:29.446 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:58:29.446 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:58:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T13:58:29.486 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:58:29.487 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:58:29.641 INFO:teuthology.misc:Compressing logs...
2023-01-14T13:58:29.669 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:58:29.707 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/log/ceph’: No such file or directory
2023-01-14T13:58:29.738 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:58:29.738 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:58:29.776 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T13:58:29.777 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T13:58:29.838 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T13:58:30.564 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:00:30.618 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:00:30.680 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:00:31.421 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:00:31.421 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:00:31.422 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:00:31.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:00:31.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:00:31.625 INFO:teuthology.orchestra.run.smithi052.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:00:31.806 INFO:teuthology.orchestra.run.smithi057.stderr:find: ‘/var/lib/ceph/crash’: No such file or directory
2023-01-14T14:00:31.834 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:00:31.835 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:00:31.875 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T14:00:31.876 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T14:00:31.877 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:00:31.906 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:00:31.920 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:00:31.983 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:00:32.697 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:02:32.752 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:02:32.852 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:02:33.618 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:02:33.618 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:02:33.619 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:02:33.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:02:33.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:02:34.084 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:02:34.113 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:02:34.125 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:02:34.188 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:02:34.887 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:04:34.942 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:04:35.001 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:04:35.707 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:04:35.707 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:04:35.708 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:04:35.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:04:35.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:04:36.146 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:04:36.184 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:04:36.189 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:04:36.251 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:04:36.986 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:06:37.035 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:06:37.090 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:06:37.807 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:06:37.808 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:06:37.808 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:06:37.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:06:37.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:06:38.297 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:06:38.328 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:06:38.337 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:06:38.392 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:06:39.077 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:08:39.132 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:08:39.197 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:08:39.945 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:08:39.946 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:08:39.946 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:08:39.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:08:40.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:08:40.463 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:08:40.494 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:08:40.496 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:08:40.559 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:08:41.284 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:10:41.342 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:10:41.404 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:10:42.134 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:10:42.135 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:10:42.135 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:10:42.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:10:42.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:10:42.653 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:10:42.685 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:10:42.686 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:10:42.757 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:10:43.468 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:12:43.522 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:12:43.586 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:12:44.363 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:12:44.364 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:12:44.364 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:12:44.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:12:44.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:12:44.854 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:12:44.884 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:12:44.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:12:44.956 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:12:45.648 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:14:45.700 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:14:45.763 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:14:46.498 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:14:46.499 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:14:46.499 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:14:46.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:14:46.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:14:47.001 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:14:47.030 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:14:47.036 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:14:47.096 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:14:47.787 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:16:47.855 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:16:47.923 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:16:48.791 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:16:48.792 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:16:48.792 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:16:48.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:16:48.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:16:49.280 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:16:49.310 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:16:49.312 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:16:49.375 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:16:50.107 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:18:50.164 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:18:50.232 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:18:50.960 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:18:50.961 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:18:50.961 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:18:50.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:18:51.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:18:51.450 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:18:51.481 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:18:51.484 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:18:51.545 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:18:52.201 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:20:52.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:20:52.313 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:20:53.033 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:20:53.033 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:20:53.034 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:20:53.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:20:53.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:20:53.534 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:20:53.563 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:20:53.565 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:20:53.623 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:20:54.278 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:22:54.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:22:54.389 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:22:55.144 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:22:55.144 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:22:55.145 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:22:55.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:22:55.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:22:55.645 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:22:55.673 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:22:55.677 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:22:55.741 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:22:56.475 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:24:56.548 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:24:56.607 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:24:57.294 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:24:57.294 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:24:57.295 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:24:57.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:24:57.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:24:57.791 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:24:57.821 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:24:57.825 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:24:57.886 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:24:58.610 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:26:58.660 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:26:58.725 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:26:59.495 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:26:59.495 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:26:59.496 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:26:59.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:26:59.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:26:59.943 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:26:59.972 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:26:59.976 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:27:00.037 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:27:00.805 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:29:00.883 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:29:00.941 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:29:01.666 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:29:01.667 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:29:01.667 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:29:01.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:29:01.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:29:02.144 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:29:02.174 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:29:02.177 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:29:02.240 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:29:02.951 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:31:03.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:31:03.084 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:31:03.829 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:31:03.829 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:31:03.830 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:31:03.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:31:04.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:31:04.336 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:31:04.366 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:31:04.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:31:04.467 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:31:05.216 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:33:05.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:33:05.333 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:33:06.074 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:33:06.074 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:33:06.075 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:33:06.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:33:06.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:33:06.529 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:33:06.559 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:33:06.563 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:33:06.630 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:33:07.321 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:35:07.375 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:35:07.440 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:35:08.266 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:35:08.267 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:35:08.267 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:35:08.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:35:08.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:35:08.748 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:35:08.777 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:35:08.781 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:35:08.844 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:35:09.614 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:37:09.667 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:37:09.723 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:37:10.416 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:37:10.417 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:37:10.417 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:37:10.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:37:10.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:37:10.899 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:37:10.932 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:37:10.932 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:37:10.934 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:37:10.994 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:37:11.711 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:39:11.775 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:39:11.836 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:39:12.559 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:39:12.559 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:39:12.559 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:39:12.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:39:12.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:39:13.089 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:39:13.124 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:39:13.124 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:39:13.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:39:13.189 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:39:13.876 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:41:13.927 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:41:13.985 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:41:14.697 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:41:14.698 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:41:14.698 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:41:14.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:41:14.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:41:15.177 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:41:15.206 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:41:15.206 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:41:15.210 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:41:15.266 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:41:15.974 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:43:16.026 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:43:16.094 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:43:16.894 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:43:16.895 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:43:16.895 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:43:16.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:43:17.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:43:17.384 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:43:17.414 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:43:17.414 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:43:17.421 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:43:17.484 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:43:18.216 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:45:18.278 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:45:18.341 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:45:19.133 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:45:19.134 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:45:19.134 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:45:19.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:45:19.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:45:19.634 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:45:19.665 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:45:19.665 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:45:19.666 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:45:19.730 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:45:20.439 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:47:20.490 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:47:20.604 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:47:21.352 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:47:21.352 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:47:21.353 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:47:21.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:47:21.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:47:21.985 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:47:22.015 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:47:22.015 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:47:22.016 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:47:22.079 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:47:22.813 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:49:22.870 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:49:22.935 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:49:23.694 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:49:23.694 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:49:23.695 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:49:23.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:49:23.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:49:24.507 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:49:24.544 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:49:24.545 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:49:24.546 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:49:24.624 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:49:25.341 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:51:25.397 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:51:25.462 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:51:26.240 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:51:26.241 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:51:26.241 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:51:26.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:51:26.420 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:51:26.883 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:51:26.916 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:51:26.916 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:51:26.917 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:51:26.974 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:51:27.667 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:53:27.721 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:53:27.788 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:53:28.545 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:53:28.546 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:53:28.546 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:53:28.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:53:28.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:53:28.993 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:53:29.022 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:53:29.022 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:53:29.026 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:53:29.109 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:53:29.868 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:55:29.928 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:55:30.017 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:55:30.776 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:55:30.777 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:55:30.777 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:55:30.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:55:30.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T14:55:31.246 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:55:31.278 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-01-14T14:55:31.279 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/kern.log.gz already exists;	not overwritten
2023-01-14T14:55:31.280 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1377, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-01-14T14:55:31.347 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:55:32.057 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:57:32.109 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:57:32.191 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:57:32.952 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:57:32.953 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:57:32.953 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:57:32.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:57:51.308 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi052
2023-01-14T14:57:51.370 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:57:52.098 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T14:59:52.149 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T14:59:52.209 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T14:59:52.964 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T14:59:52.965 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T14:59:52.965 INFO:teuthology.misc:Compressing logs...
2023-01-14T14:59:52.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T14:59:53.082 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T14:59:54.199 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T14:59:55.316 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T14:59:56.392 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T14:59:57.508 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T14:59:58.624 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T14:59:59.740 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:00.855 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:01.970 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:03.085 ERROR:teuthology.orchestra.connection:Error authenticating with smithi052.front.sepia.ceph.com: Authentication failed.
2023-01-14T15:00:03.085 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi052
2023-01-14T15:00:03.147 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:00:03.889 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:02:03.948 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:02:04.009 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:02:04.789 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:02:04.790 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T15:02:04.790 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:02:04.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T15:02:07.852 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi052
2023-01-14T15:02:07.910 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:02:08.601 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:04:08.649 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:04:08.717 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:04:09.495 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:04:09.496 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T15:04:09.496 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:04:09.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T15:04:09.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T15:04:09.736 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:04:10.547 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:04:10.583 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:04:10.583 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:10.627 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:04:10.627 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:10.629 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:04:10.645 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-14T15:04:10.688 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/home/ubuntu/cephtest/archive': No such file or directory
2023-01-14T15:04:10.707 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T15:04:10.707 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:10.723 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory
2023-01-14T15:04:10.723 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:10.726 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:04:10.741 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:04:10.741 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:04:10.805 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:04:10.806 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:10.818 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:04:10.818 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:04:10.881 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:04:11.639 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:06:11.691 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:06:11.755 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:06:12.627 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:06:12.627 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T15:06:12.628 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:06:12.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T15:06:44.048 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 289, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/dispatcher/supervisor.py", line 348, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/misc.py", line 1378, in compress_logs
    ctx.cluster.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in run
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/cluster.py", line 85, in <listcomp>
    procs = [remote.run(**kwargs, wait=_wait) for remote in remotes]
  File "/home/teuthworker/src/git.ceph.com_teuthology_19d18db866afdc31fd6586f558fc29b95a87ccfb/teuthology/orchestra/remote.py", line 524, in run
    raise Exception(f'Cannot connect to remote host {self.shortname}')
Exception: Cannot connect to remote host smithi052
2023-01-14T15:06:44.115 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:06:44.894 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:08:44.946 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:08:45.011 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:08:45.783 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:08:45.784 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T15:08:45.784 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:08:45.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T15:08:45.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T15:08:45.974 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:08:46.151 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:08:46.180 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:08:46.180 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:08:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:08:46.202 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:08:46.205 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:08:46.337 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:08:46.363 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:08:46.365 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:08:46.380 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:08:46.381 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:08:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:08:46.422 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:08:46.486 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:08:47.217 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:10:47.276 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:10:47.344 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:10:48.118 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:10:48.118 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T15:10:48.119 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:10:48.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T15:10:48.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T15:10:48.312 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:10:48.485 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:10:48.504 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:10:48.505 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:10:48.544 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:10:48.544 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:10:48.546 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:10:48.675 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:10:48.692 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:10:48.735 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:10:48.756 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:10:48.757 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:10:48.774 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:10:48.775 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:10:48.840 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:10:49.603 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:12:49.661 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:12:49.723 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:12:50.503 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:12:50.503 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T15:12:50.504 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:12:50.504 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T15:12:50.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T15:12:50.694 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:12:50.864 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/lib/ceph/crash': No such file or directory
2023-01-14T15:12:50.883 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:12:50.883 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:12:50.900 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/lib/ceph/crash: Cannot open: No such file or directory
2023-01-14T15:12:50.900 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:12:50.902 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:12:51.028 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:12:51.042 INFO:teuthology.orchestra.run.smithi052.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:12:51.044 INFO:teuthology.orchestra.run.smithi057.stderr:find: '/var/log/ceph': No such file or directory
2023-01-14T15:12:51.060 INFO:teuthology.orchestra.run.smithi052.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:12:51.061 INFO:teuthology.orchestra.run.smithi052.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:12:51.104 INFO:teuthology.orchestra.run.smithi057.stderr:tar: /var/log/ceph: Cannot open: No such file or directory
2023-01-14T15:12:51.105 INFO:teuthology.orchestra.run.smithi057.stderr:tar: Error is not recoverable: exiting now
2023-01-14T15:12:51.168 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:12:51.945 INFO:teuthology.kill:No locked machines. Not nuking anything
2023-01-14T15:14:51.995 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-14T15:14:52.058 INFO:teuthology.kill:Killing Pids: {25351}
2023-01-14T15:14:52.848 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-01-14T15:14:52.848 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
2023-01-14T15:14:52.849 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:14:52.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com'
2023-01-14T15:14:53.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2023-01-14T15:14:53.298 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:14:53.419 INFO:teuthology.misc:Compressing logs...
2023-01-14T15:14:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping