2023-03-16T08:08:44.657 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f
2023-03-16T08:08:47.380 INFO:teuthology.lock.ops:Start node 'smithi045.front.sepia.ceph.com' reimaging
2023-03-16T08:08:48.969 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T08:08:51.855 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2023-03-16T08:08:51.889 INFO:teuthology.lock.ops:Node 'smithi045.front.sepia.ceph.com' reimaging is complete
2023-03-16T08:08:53.125 INFO:teuthology.lock.ops:Start node 'smithi082.front.sepia.ceph.com' reimaging
2023-03-16T08:08:53.125 INFO:teuthology.lock.ops:Updating [smithi082.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T08:08:53.125 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server
2023-03-16T08:08:53.164 INFO:teuthology.lock.ops:Node 'smithi082.front.sepia.ceph.com' reimaging is complete
2023-03-16T08:08:54.466 INFO:teuthology.lock.ops:Start node 'smithi142.front.sepia.ceph.com' reimaging
2023-03-16T08:08:54.466 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T08:08:54.466 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2023-03-16T08:08:54.488 INFO:teuthology.lock.ops:Node 'smithi142.front.sepia.ceph.com' reimaging is complete
2023-03-16T08:08:56.588 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of rhel 8.6
2023-03-16T08:08:58.864 INFO:teuthology.provision.fog.smithi082:Scheduling deploy of rhel 8.6
2023-03-16T08:09:00.439 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of rhel 8.6
2023-03-16T08:09:01.536 INFO:teuthology.orchestra.console:Power off smithi045
2023-03-16T08:09:02.961 INFO:teuthology.orchestra.console:Power off smithi082
2023-03-16T08:09:04.510 INFO:teuthology.orchestra.console:Power off smithi142
2023-03-16T08:09:05.483 INFO:teuthology.orchestra.console:Power off for smithi045 completed
2023-03-16T08:09:06.861 INFO:teuthology.orchestra.console:Power on smithi045
2023-03-16T08:09:07.675 INFO:teuthology.orchestra.console:Power off for smithi082 completed
2023-03-16T08:09:09.017 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2023-03-16T08:09:09.118 INFO:teuthology.orchestra.console:Power on smithi082
2023-03-16T08:09:09.946 INFO:teuthology.orchestra.console:Power on smithi142
2023-03-16T08:09:16.649 INFO:teuthology.orchestra.console:Power on for smithi082 completed
2023-03-16T08:09:18.150 INFO:teuthology.orchestra.console:Power on for smithi045 completed
2023-03-16T08:09:18.251 INFO:teuthology.provision.fog.smithi082:Waiting for deploy to finish
2023-03-16T08:09:19.357 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish
2023-03-16T08:09:24.879 INFO:teuthology.orchestra.console:Power on for smithi142 completed
2023-03-16T08:09:29.767 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish
2023-03-16T08:11:53.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-03-16T08:11:55.728 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-16T08:12:04.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-03-16T08:12:16.305 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-16T08:12:18.557 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-03-16T08:12:25.385 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.45
2023-03-16T08:12:28.922 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-16T08:12:31.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-03-16T08:12:37.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.82
2023-03-16T08:12:42.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-03-16T08:12:52.421 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.142
2023-03-16T08:13:34.925 WARNING:teuthology.provision.fog:timed out
2023-03-16T08:13:39.026 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T08:13:39.746 INFO:teuthology.provision.fog.smithi142:Node is ready
2023-03-16T08:13:40.973 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142.front.sepia.ceph.com
2023-03-16T08:13:41.950 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142
2023-03-16T08:13:43.244 INFO:teuthology.provision.fog.smithi142:Deploy complete!
2023-03-16T08:13:44.309 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T08:13:46.607 WARNING:teuthology.provision.fog:timed out
2023-03-16T08:13:46.699 INFO:teuthology.provision.fog.smithi045:Node is ready
2023-03-16T08:13:47.803 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045.front.sepia.ceph.com
2023-03-16T08:13:49.178 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45 smithi045.front.sepia.ceph.com smithi045
2023-03-16T08:13:51.122 INFO:teuthology.provision.fog.smithi045:Deploy complete!
2023-03-16T08:13:52.956 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T08:13:54.485 INFO:teuthology.provision.fog.smithi082:Node is ready
2023-03-16T08:13:56.080 INFO:teuthology.orchestra.run.smithi082.stdout:smithi082.front.sepia.ceph.com
2023-03-16T08:13:57.328 INFO:teuthology.orchestra.run.smithi082.stdout:172.21.15.82 smithi082.front.sepia.ceph.com smithi082
2023-03-16T08:13:59.557 INFO:teuthology.provision.fog.smithi082:Deploy complete!
2023-03-16T08:14:01.050 INFO:teuthology.lock.ops:Checking smithi045.front.sepia.ceph.com
2023-03-16T08:14:02.128 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T08:14:02.153 INFO:teuthology.lock.ops:Checking smithi082.front.sepia.ceph.com
2023-03-16T08:14:02.853 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T08:14:02.878 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com
2023-03-16T08:14:03.832 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T08:14:03.854 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: set os type and version on server
2023-03-16T08:14:04.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T08:14:04.514 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64
2023-03-16T08:14:04.573 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="Red Hat Enterprise Linux"
2023-03-16T08:14:05.014 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8.6 (Ootpa)"
2023-03-16T08:14:05.014 INFO:teuthology.orchestra.run.smithi045.stdout:ID="rhel"
2023-03-16T08:14:05.014 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="fedora"
2023-03-16T08:14:05.014 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8.6"
2023-03-16T08:14:05.014 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8"
2023-03-16T08:14:05.014 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-16T08:14:05.014 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31"
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://www.redhat.com/"
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-16T08:14:05.015 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-16T08:14:05.499 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2023-03-16T08:14:06.311 INFO:teuthology.lock.ops:Updating [smithi082.front.sepia.ceph.com]: set os type and version on server
2023-03-16T08:14:07.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T08:14:07.295 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64
2023-03-16T08:14:08.169 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="Red Hat Enterprise Linux"
2023-03-16T08:14:09.316 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="8.6 (Ootpa)"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:ID="rhel"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="fedora"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="8.6"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el8"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://www.redhat.com/"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-16T08:14:09.317 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-16T08:14:09.318 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server
2023-03-16T08:14:09.344 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server
2023-03-16T08:14:10.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T08:14:11.479 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64
2023-03-16T08:14:12.132 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Red Hat Enterprise Linux"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8.6 (Ootpa)"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:ID="rhel"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="fedora"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8.6"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.redhat.com/"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:
2023-03-16T08:14:13.411 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-16T08:14:13.412 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-16T08:14:13.412 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-16T08:14:13.412 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-16T08:14:13.412 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2023-03-16T08:14:13.536 INFO:teuthology.dispatcher.supervisor:Running job 7210186
2023-03-16T08:14:14.358 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2023-03-16_07:05:31-fs:workload-wip-20230316-1205-unlink-distro-default-smithi/7210186 --name xiubli-2023-03-16_07:05:31-fs:workload-wip-20230316-1205-unlink-distro-default-smithi --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} -- /home/teuthworker/archive/xiubli-2023-03-16_07:05:31-fs:workload-wip-20230316-1205-unlink-distro-default-smithi/7210186/orig.config.yaml
2023-03-16T08:14:14.362 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2023-03-16_07:05:31-fs:workload-wip-20230316-1205-unlink-distro-default-smithi/7210186
2023-03-16T08:14:15.321 INFO:teuthology.dispatcher.supervisor:Job PID: 29820
2023-03-16T08:14:15.322 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-16T20:15:07.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:15:08.554 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:15:09.494 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
KeyError: 'status'
2023-03-16T20:15:10.305 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:15:11.274 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:15:11.274 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:15:11.275 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:15:11.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:15:11.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:15:13.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:15:14.730 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:15:15.724 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:15:31.955 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2023-03-16T20:15:40.992 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-03-16T20:15:43.890 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-16T20:15:43.890 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mds.h.log: file size changed while zipping
2023-03-16T20:15:44.018 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mds.e.log: file size changed while zipping
2023-03-16T20:15:51.501 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2023-03-16T20:15:55.964 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2023-03-16T20:15:59.976 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-03-16T20:16:09.054 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-16T20:16:11.106 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2023-03-16T20:16:14.982 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2023-03-16T20:16:15.988 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mgr.x.log: file size changed while zipping
2023-03-16T20:16:24.621 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-16T20:16:27.053 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mds.f.log: file size changed while zipping
2023-03-16T20:16:30.613 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.9.log: file size changed while zipping
2023-03-16T20:16:32.579 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mds.c.log: file size changed while zipping
2023-03-16T20:16:36.512 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-16T20:16:36.513 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mds.i.log: file size changed while zipping
2023-03-16T20:16:40.191 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2023-03-16T20:16:41.678 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mds.b.log: file size changed while zipping
2023-03-16T20:16:45.233 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2023-03-16T20:16:59.178 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2023-03-16T20:18:00.655 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:18:02.090 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:20:06.079 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:20:07.210 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:20:08.677 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:20:10.376 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:20:12.709 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:20:12.709 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:20:12.709 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:20:12.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:20:12.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:20:13.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:20:15.365 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:20:17.948 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:20:20.417 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:20:22.783 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:20:22.783 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:20:22.784 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:20:22.854 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:20:23.910 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:22:25.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:22:25.995 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:22:27.523 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:22:28.495 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:22:30.484 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:22:30.485 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:22:31.092 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:22:31.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:22:31.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:22:32.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:22:33.591 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:22:34.380 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:22:34.984 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:22:36.131 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:22:38.701 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:22:38.702 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:22:38.769 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:22:40.391 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:24:44.625 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:24:46.757 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:24:48.049 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:24:48.959 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:24:49.924 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:24:49.924 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:24:49.925 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:24:49.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:24:50.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:24:51.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:24:54.209 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:24:55.261 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:24:59.618 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:25:01.802 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:25:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:25:01.802 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:25:01.803 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:25:01.869 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:25:03.297 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:27:04.730 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:27:05.760 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:27:07.883 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:27:09.850 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:27:12.229 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:27:12.229 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:27:12.229 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:27:12.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:27:12.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:27:15.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:27:18.211 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:27:22.225 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:27:24.736 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:27:27.807 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:27:27.809 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:27:28.789 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:27:28.789 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:27:30.917 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:27:30.918 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:27:30.986 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:27:33.478 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:29:34.637 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:29:35.926 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:29:38.964 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:29:41.490 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:29:43.563 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:29:43.563 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:29:43.563 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:29:43.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:29:43.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:29:44.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:29:46.217 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:29:47.278 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:29:47.954 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:29:48.259 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:29:48.261 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:29:49.718 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:29:49.718 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:29:49.718 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:29:49.718 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:29:49.786 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:29:50.909 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:31:51.683 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:31:52.892 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:31:53.723 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:31:54.455 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:31:56.321 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:31:56.321 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:31:56.321 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:31:56.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:31:56.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:31:58.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:32:00.501 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:32:01.617 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:32:04.110 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:32:07.328 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:32:07.330 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:32:07.330 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:32:07.330 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:32:07.330 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:32:07.331 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:32:07.397 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:32:09.129 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:34:11.733 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:34:12.945 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:34:13.911 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:34:15.371 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:34:17.561 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:34:17.562 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:34:17.562 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:34:17.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:34:18.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:34:20.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:34:23.383 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:34:25.771 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:34:27.464 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:34:29.320 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:34:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:34:29.322 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:34:29.323 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:34:29.323 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:34:29.323 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:34:29.390 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:34:30.842 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:36:32.241 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:36:34.539 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:36:38.292 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:36:40.452 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:36:42.607 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:36:42.607 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:36:42.607 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:36:42.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:36:42.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:36:45.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:36:46.977 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:36:48.338 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:36:48.647 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:36:49.622 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:36:49.624 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:36:50.968 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:36:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:36:50.968 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:36:50.968 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:36:52.803 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:36:55.477 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:38:57.037 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:38:58.100 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:38:58.927 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:38:59.859 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:39:00.967 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:39:00.967 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:39:00.967 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:39:00.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:39:01.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:39:03.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:39:05.803 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:39:07.015 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:39:07.991 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:39:10.205 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:39:10.207 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:39:10.208 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:39:10.208 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:39:10.209 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:39:10.209 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:39:12.321 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:39:13.651 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:41:14.405 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:41:15.954 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:41:19.009 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:41:21.291 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:41:22.382 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:41:22.382 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:41:23.001 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:41:23.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:41:23.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:41:24.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:41:26.294 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:41:27.176 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:41:28.695 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:41:29.210 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:41:30.848 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:41:33.856 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:41:33.856 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:41:33.856 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:41:33.856 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:41:35.288 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:41:37.337 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:43:39.333 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:43:39.799 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:43:41.847 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:43:43.276 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:43:44.284 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:43:45.829 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:43:45.829 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:43:45.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:43:46.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:43:47.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:43:48.288 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:43:49.968 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:43:51.361 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:43:51.940 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:43:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:43:51.942 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:43:51.942 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:43:52.832 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:43:52.833 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:43:52.901 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:43:53.658 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:45:55.039 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T20:45:55.372 INFO:teuthology.kill:Killing Pids: {29820}
2023-03-16T20:45:56.334 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:45:57.436 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T20:45:59.765 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T20:45:59.765 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T20:45:59.765 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:45:59.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2023-03-16T20:45:59.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com'
2023-03-16T20:46:01.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2023-03-16T20:46:02.616 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:46:03.455 INFO:teuthology.misc:Compressing logs...
2023-03-16T20:46:04.765 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:46:05.234 INFO:teuthology.orchestra.run.smithi082.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:46:06.571 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:46:07.010 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:46:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T20:46:07.897 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/d9c79c4c-c3d6-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T20:46:07.897 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi045 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T20:46:07.965 INFO:teuthology.kill:No teuthology processes running
2023-03-16T20:46:08.243 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-16T20:48:08.473 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-16T20:48:09.955 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-16T20:48:10.484 INFO:teuthology.nuke:Checking targets against current locks
2023-03-16T20:48:10.543 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-16T20:48:12.089 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-16T20:48:13.741 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-16T20:48:16.247 INFO:teuthology.orchestra.console:Power off smithi045
2023-03-16T20:48:18.319 INFO:teuthology.orchestra.console:Power off smithi142
2023-03-16T20:48:20.104 INFO:teuthology.orchestra.console:Power off smithi082
2023-03-16T20:48:30.006 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2023-03-16T20:48:32.486 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com
2023-03-16T20:48:33.676 INFO:teuthology.orchestra.console:Power off for smithi045 completed
2023-03-16T20:48:33.836 INFO:teuthology.lock.ops:unlocked: smithi045.front.sepia.ceph.com
2023-03-16T20:48:35.148 INFO:teuthology.orchestra.console:Power off for smithi082 completed
2023-03-16T20:48:36.610 INFO:teuthology.lock.ops:unlocked: smithi082.front.sepia.ceph.com