2023-03-17T20:22:23.773 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3
2023-03-17T20:22:25.326 INFO:teuthology.lock.ops:Start node 'smithi032.front.sepia.ceph.com' reimaging
2023-03-17T20:22:26.241 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: reset os type and version on server
2023-03-17T20:22:26.241 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2023-03-17T20:22:26.268 INFO:teuthology.lock.ops:Node 'smithi032.front.sepia.ceph.com' reimaging is complete
2023-03-17T20:22:27.856 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging
2023-03-17T20:22:27.856 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server
2023-03-17T20:22:27.856 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server
2023-03-17T20:22:27.884 INFO:teuthology.lock.ops:Node 'smithi053.front.sepia.ceph.com' reimaging is complete
2023-03-17T20:22:29.383 INFO:teuthology.lock.ops:Start node 'smithi114.front.sepia.ceph.com' reimaging
2023-03-17T20:22:29.383 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: reset os type and version on server
2023-03-17T20:22:29.383 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2023-03-17T20:22:29.418 INFO:teuthology.lock.ops:Node 'smithi114.front.sepia.ceph.com' reimaging is complete
2023-03-17T20:22:31.540 INFO:teuthology.provision.fog.smithi032:Scheduling deploy of rhel 8.4
2023-03-17T20:22:33.711 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of rhel 8.4
2023-03-17T20:22:33.758 INFO:teuthology.provision.fog.smithi114:Scheduling deploy of rhel 8.4
2023-03-17T20:22:37.703 INFO:teuthology.orchestra.console:Power off smithi053
2023-03-17T20:22:39.388 INFO:teuthology.orchestra.console:Power off smithi032
2023-03-17T20:22:40.965 INFO:teuthology.orchestra.console:Power off for smithi053 completed
2023-03-17T20:22:43.335 INFO:teuthology.orchestra.console:Power off smithi114
2023-03-17T20:22:44.173 INFO:teuthology.orchestra.console:Power on smithi053
2023-03-17T20:22:45.163 INFO:teuthology.orchestra.console:Power off for smithi032 completed
2023-03-17T20:22:45.823 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2023-03-17T20:22:45.923 INFO:teuthology.orchestra.console:Power on smithi032
2023-03-17T20:22:46.624 INFO:teuthology.orchestra.console:Power on smithi114
2023-03-17T20:22:49.283 INFO:teuthology.orchestra.console:Power on for smithi053 completed
2023-03-17T20:22:50.316 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish
2023-03-17T20:22:52.177 INFO:teuthology.orchestra.console:Power on for smithi032 completed
2023-03-17T20:22:54.465 INFO:teuthology.provision.fog.smithi032:Waiting for deploy to finish
2023-03-17T20:22:56.294 INFO:teuthology.orchestra.console:Power on for smithi114 completed
2023-03-17T20:22:58.342 INFO:teuthology.provision.fog.smithi114:Waiting for deploy to finish
2023-03-17T20:25:24.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53
2023-03-17T20:25:29.573 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-03-17T20:25:35.301 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53
2023-03-17T20:25:43.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-03-17T20:25:49.033 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53
2023-03-17T20:25:54.416 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.114
2023-03-17T20:25:57.477 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.32
2023-03-17T20:26:03.493 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.114
2023-03-17T20:26:14.505 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.114
2023-03-17T20:26:25.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.114
2023-03-17T20:27:00.421 WARNING:teuthology.provision.fog:timed out
2023-03-17T20:27:05.454 WARNING:teuthology.provision.fog:timed out
2023-03-17T20:27:07.909 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-17T20:27:09.839 INFO:teuthology.provision.fog.smithi053:Node is ready
2023-03-17T20:27:11.840 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053.front.sepia.ceph.com
2023-03-17T20:27:13.076 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53 smithi053.front.sepia.ceph.com smithi053
2023-03-17T20:27:13.623 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-17T20:27:14.125 INFO:teuthology.provision.fog.smithi032:Node is ready
2023-03-17T20:27:16.639 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032.front.sepia.ceph.com
2023-03-17T20:27:19.362 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32 smithi032.front.sepia.ceph.com smithi032
2023-03-17T20:27:22.097 INFO:teuthology.provision.fog.smithi053:Deploy complete!
2023-03-17T20:27:23.880 INFO:teuthology.provision.fog.smithi032:Deploy complete!
2023-03-17T20:27:32.501 WARNING:teuthology.provision.fog:timed out
2023-03-17T20:27:39.724 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-17T20:27:40.943 INFO:teuthology.provision.fog.smithi114:Node is ready
2023-03-17T20:27:42.366 INFO:teuthology.orchestra.run.smithi114.stdout:smithi114.front.sepia.ceph.com
2023-03-17T20:27:44.903 INFO:teuthology.orchestra.run.smithi114.stdout:172.21.15.114 smithi114.front.sepia.ceph.com smithi114
2023-03-17T20:27:47.543 INFO:teuthology.provision.fog.smithi114:Deploy complete!
2023-03-17T20:27:47.899 INFO:teuthology.lock.ops:Checking smithi032.front.sepia.ceph.com
2023-03-17T20:27:48.884 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-17T20:27:48.916 INFO:teuthology.lock.ops:Checking smithi053.front.sepia.ceph.com
2023-03-17T20:27:50.121 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-17T20:27:50.140 INFO:teuthology.lock.ops:Checking smithi114.front.sepia.ceph.com
2023-03-17T20:27:51.796 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-17T20:27:51.825 INFO:teuthology.lock.ops:Updating [smithi032.front.sepia.ceph.com]: set os type and version on server
2023-03-17T20:27:52.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-17T20:27:53.078 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64
2023-03-17T20:27:53.797 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="Red Hat Enterprise Linux"
2023-03-17T20:27:54.653 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="8.4 (Ootpa)"
2023-03-17T20:27:54.653 INFO:teuthology.orchestra.run.smithi032.stdout:ID="rhel"
2023-03-17T20:27:54.653 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="fedora"
2023-03-17T20:27:54.653 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="8.4"
2023-03-17T20:27:54.653 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el8"
2023-03-17T20:27:54.653 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-17T20:27:56.388 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31"
2023-03-17T20:27:56.388 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-17T20:27:56.388 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://www.redhat.com/"
2023-03-17T20:27:56.389 INFO:teuthology.orchestra.run.smithi032.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-17T20:27:56.389 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-17T20:27:56.389 INFO:teuthology.orchestra.run.smithi032.stdout:
2023-03-17T20:27:56.389 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-17T20:27:56.389 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-17T20:27:56.389 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-17T20:27:56.389 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-17T20:27:56.390 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server
2023-03-17T20:27:56.414 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: set os type and version on server
2023-03-17T20:27:57.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-17T20:27:57.658 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64
2023-03-17T20:27:59.739 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="Red Hat Enterprise Linux"
2023-03-17T20:28:02.211 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8.4 (Ootpa)"
2023-03-17T20:28:02.211 INFO:teuthology.orchestra.run.smithi053.stdout:ID="rhel"
2023-03-17T20:28:02.211 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="fedora"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8.4"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://www.redhat.com/"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-17T20:28:02.212 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-17T20:28:02.213 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server
2023-03-17T20:28:02.247 INFO:teuthology.lock.ops:Updating [smithi114.front.sepia.ceph.com]: set os type and version on server
2023-03-17T20:28:04.041 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-17T20:28:04.274 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64
2023-03-17T20:28:05.925 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="Red Hat Enterprise Linux"
2023-03-17T20:28:07.946 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8.4 (Ootpa)"
2023-03-17T20:28:07.946 INFO:teuthology.orchestra.run.smithi114.stdout:ID="rhel"
2023-03-17T20:28:07.946 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="fedora"
2023-03-17T20:28:07.946 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8.4"
2023-03-17T20:28:07.946 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://www.redhat.com/"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-17T20:28:07.947 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-17T20:28:07.948 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server
2023-03-17T20:28:08.071 INFO:teuthology.dispatcher.supervisor:Running job 7211828
2023-03-17T20:28:08.768 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2023-03-17_17:02:43-fs-quincy-release-testing-default-smithi/7211828 --name vshankar-2023-03-17_17:02:43-fs-quincy-release-testing-default-smithi --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} -- /home/teuthworker/archive/vshankar-2023-03-17_17:02:43-fs-quincy-release-testing-default-smithi/7211828/orig.config.yaml
2023-03-17T20:28:08.773 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2023-03-17_17:02:43-fs-quincy-release-testing-default-smithi/7211828
2023-03-17T20:28:11.060 INFO:teuthology.dispatcher.supervisor:Job PID: 9595
2023-03-17T20:28:11.060 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-18T08:29:05.209 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:29:06.115 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:29:08.428 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:29:08.948 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:29:09.992 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:29:09.992 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:29:09.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:29:10.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:29:11.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:29:12.596 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:29:14.360 INFO:teuthology.orchestra.run.smithi032.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-18T08:29:15.433 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-18T08:29:16.464 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-18T08:29:17.043 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:29:22.147 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-18T08:29:23.411 INFO:teuthology.orchestra.run.smithi032.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.9.log: file size changed while zipping
2023-03-18T08:29:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-18T08:29:25.052 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-18T08:29:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-18T08:29:26.890 INFO:teuthology.orchestra.run.smithi032.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2023-03-18T08:29:27.927 INFO:teuthology.orchestra.run.smithi114.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-03-18T08:29:30.145 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /var/log/ceph/a7458638-c504-11ed-9afc-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-03-18T08:29:48.043 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:29:48.627 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:31:49.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:31:50.988 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:31:52.214 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:31:53.414 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:31:54.639 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:31:54.639 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:31:54.639 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:31:55.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:31:55.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:31:56.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:31:58.063 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:31:59.558 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:32:17.170 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:32:18.653 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:34:20.117 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:34:21.974 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:34:23.379 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:34:26.891 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:34:28.601 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:34:28.601 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:34:28.601 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:34:28.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:34:28.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:34:29.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:34:30.417 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:34:32.469 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:34:50.383 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:34:52.097 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:36:53.187 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:36:55.256 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:36:56.819 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:36:57.682 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:36:58.999 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:36:58.999 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:36:59.000 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:36:59.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:36:59.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:36:59.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:37:01.713 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:37:03.973 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:37:21.518 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:37:22.981 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:39:25.389 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:39:26.864 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:39:28.618 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:39:30.424 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:39:32.195 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:39:32.195 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:39:32.195 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:39:32.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:39:32.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:39:34.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:39:35.892 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:39:38.224 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:40:01.232 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:40:02.712 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:42:04.236 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:42:06.560 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:42:08.118 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:42:09.653 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:42:12.478 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:42:12.478 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:42:14.748 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:42:14.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:42:14.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:42:18.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:42:20.650 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:42:22.178 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:42:39.194 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:42:40.996 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:44:42.221 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:44:44.475 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:44:46.944 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:44:47.986 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:44:48.740 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:44:48.740 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:44:48.741 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:44:48.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:44:48.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:44:51.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:44:53.877 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:44:55.069 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:45:13.621 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:45:14.724 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:47:16.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:47:19.081 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:47:20.458 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:47:22.075 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:47:23.620 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:47:23.620 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:47:23.621 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:47:23.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:47:23.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:47:24.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:47:25.225 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:47:26.600 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:47:43.883 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:47:46.617 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:49:50.499 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:49:50.959 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:49:51.521 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:49:52.361 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:49:53.574 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:49:53.574 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:49:53.574 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:49:53.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:49:53.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:49:55.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:49:56.692 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:49:58.706 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:50:17.330 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:50:17.956 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:52:19.490 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:52:20.190 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:52:21.212 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:52:22.344 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:52:23.903 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:52:23.903 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:52:23.903 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:52:23.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:52:25.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:52:25.997 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:52:27.285 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:52:28.998 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:52:49.357 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:52:51.056 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:54:52.457 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:54:53.599 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:54:55.048 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:54:56.824 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:54:57.364 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:54:57.364 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:54:57.364 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:54:57.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:54:57.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:54:58.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:55:02.199 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:55:03.896 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:55:23.620 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:55:24.582 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:57:26.722 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T08:57:27.278 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:57:35.015 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T08:57:39.561 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T08:57:40.982 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T08:57:40.982 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T08:57:40.982 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:57:40.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T08:57:41.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T08:57:41.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T08:57:53.013 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:57:55.389 INFO:teuthology.misc:Compressing logs...
2023-03-18T08:58:12.103 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T08:58:13.688 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:00:15.066 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:00:16.036 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:00:16.595 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:00:18.437 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:00:19.742 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:00:20.763 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:00:20.763 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:00:20.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:00:20.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:00:21.461 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:00:22.460 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:00:24.896 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:00:42.123 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:00:42.799 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:02:44.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:02:45.741 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:02:47.121 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:02:48.313 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:02:50.574 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:02:50.575 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:02:50.575 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:02:50.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:02:50.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:02:53.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:02:55.623 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:02:57.276 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:03:14.352 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:03:17.402 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:05:20.837 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:05:22.394 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:05:23.819 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:05:25.012 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:05:33.434 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:05:33.434 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:05:33.434 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:05:33.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:05:33.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:05:35.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:05:37.221 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:05:38.949 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:05:56.989 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:05:57.363 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:07:58.109 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:08:07.485 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:08:08.831 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:08:09.489 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:08:10.357 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:08:11.522 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:08:11.522 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:08:11.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:08:11.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:08:13.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:08:15.109 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:08:17.733 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:08:35.141 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:08:46.135 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:10:47.247 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:10:51.252 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:10:54.586 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:10:57.494 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:10:59.107 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:10:59.107 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:10:59.107 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:10:59.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:10:59.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:11:00.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:11:03.648 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:11:05.711 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:11:22.834 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:11:24.724 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:13:26.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:13:27.636 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:13:29.871 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:13:32.034 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:13:33.765 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:13:33.765 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:13:33.765 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:13:33.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:13:33.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:13:35.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:13:36.598 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:13:37.637 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:13:54.563 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:13:54.739 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:15:56.325 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:15:57.148 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:15:58.544 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:16:00.152 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:16:01.345 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:16:01.346 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:16:01.346 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:16:01.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:16:01.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:16:06.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:16:07.959 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:16:11.482 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:16:29.050 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:16:30.594 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:18:32.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:18:36.403 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:18:39.086 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:18:42.170 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:18:43.919 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:18:43.919 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:18:43.919 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:18:43.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:18:48.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:18:50.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:18:52.443 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:18:54.012 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:19:10.525 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:19:13.123 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:21:15.641 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:21:16.630 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:21:19.934 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:21:20.602 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:21:21.881 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:21:21.882 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:21:21.882 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:21:21.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:21:22.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:21:23.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:21:27.219 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:21:29.832 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:21:47.999 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:21:49.336 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:23:51.279 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:23:52.819 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:23:53.825 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:23:55.680 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:23:56.700 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:23:56.700 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:23:56.700 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:23:56.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:23:56.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:23:57.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:23:59.905 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:24:02.913 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:24:20.601 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:24:22.983 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:26:25.586 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:26:26.732 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:26:27.756 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:26:29.893 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:26:32.373 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:26:32.373 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:26:32.373 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:26:32.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:26:32.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:26:33.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:26:34.898 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:26:36.224 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:26:54.693 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:26:55.775 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:28:56.457 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:28:57.943 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:28:59.488 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:29:00.662 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:29:02.115 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:29:02.115 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:29:02.115 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:29:02.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:29:02.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:29:03.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:29:07.419 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:29:10.475 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:29:29.516 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:29:30.327 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:31:32.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:31:34.381 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:31:35.981 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:31:37.585 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:31:38.972 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:31:40.567 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:31:40.567 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:31:40.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:31:40.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:31:42.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:31:44.637 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:31:46.336 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:32:04.631 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:32:05.755 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:34:05.934 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:34:06.923 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:34:08.048 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:34:09.302 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:34:10.416 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:34:10.416 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:34:10.416 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:34:10.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:34:11.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:34:12.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:34:13.857 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:34:16.006 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:34:34.251 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:34:34.832 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:36:36.138 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:36:37.354 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:36:38.828 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:36:41.264 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:36:42.578 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:36:42.578 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:36:42.578 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:36:42.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:36:42.761 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:36:44.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:36:47.459 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:36:49.379 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:37:08.163 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:37:09.723 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:39:10.703 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:39:12.862 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:39:13.825 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:39:15.238 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:39:17.765 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:39:17.765 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:39:17.765 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:39:17.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:39:17.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:39:18.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:39:20.272 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:39:23.127 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:39:40.852 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:39:42.619 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:41:43.379 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:41:44.286 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:41:45.411 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:41:47.397 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:41:49.853 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:41:49.853 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:41:49.853 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:41:49.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:41:50.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:41:51.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:41:53.290 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:41:55.589 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:42:13.349 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:42:14.248 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:44:16.645 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:44:17.624 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:44:20.614 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:44:21.824 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:44:23.791 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:44:23.791 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:44:23.791 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:44:23.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:44:23.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:44:25.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:44:26.657 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:44:29.224 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:44:46.962 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:44:48.391 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:46:49.997 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:46:50.963 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:46:53.320 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:46:54.592 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:46:56.504 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:46:58.491 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:46:58.491 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:46:58.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:46:58.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:47:00.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:47:01.878 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:47:04.441 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:47:22.011 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:47:23.062 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:49:24.889 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T09:49:26.403 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:49:27.842 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:49:28.998 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-18T09:49:31.643 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-18T09:49:31.644 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-18T09:49:31.644 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:49:31.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com'
2023-03-18T09:49:31.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com'
2023-03-18T09:49:33.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com'
2023-03-18T09:49:36.638 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:49:38.267 INFO:teuthology.misc:Compressing logs...
2023-03-18T09:49:55.614 INFO:teuthology.kill:Killing Pids: {9595}
2023-03-18T09:49:57.686 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'}
2023-03-18T09:51:59.481 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-18T09:52:00.430 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-18T09:52:01.370 INFO:teuthology.nuke:Checking targets against current locks
2023-03-18T09:52:02.482 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T09:52:03.924 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T09:52:05.333 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T09:52:06.278 INFO:teuthology.orchestra.console:Power off smithi053
2023-03-18T09:52:07.155 INFO:teuthology.orchestra.console:Power off smithi032
2023-03-18T09:52:08.511 INFO:teuthology.orchestra.console:Power off smithi114
2023-03-18T09:52:16.808 INFO:teuthology.orchestra.console:Power off for smithi053 completed
2023-03-18T09:52:19.160 INFO:teuthology.lock.ops:unlocked: smithi053.front.sepia.ceph.com
2023-03-18T09:52:20.222 INFO:teuthology.orchestra.console:Power off for smithi032 completed
2023-03-18T09:52:20.223 INFO:teuthology.orchestra.console:Power off for smithi114 completed
2023-03-18T09:52:20.377 INFO:teuthology.lock.ops:unlocked: smithi032.front.sepia.ceph.com
2023-03-18T09:52:20.678 INFO:teuthology.lock.ops:unlocked: smithi114.front.sepia.ceph.com