2023-03-19T09:34:37.444 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3
2023-03-19T09:34:38.334 INFO:teuthology.lock.ops:Start node 'smithi111.front.sepia.ceph.com' reimaging
2023-03-19T09:34:40.763 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: reset os type and version on server
2023-03-19T09:34:40.763 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server
2023-03-19T09:34:40.789 INFO:teuthology.lock.ops:Node 'smithi111.front.sepia.ceph.com' reimaging is complete
2023-03-19T09:34:41.775 INFO:teuthology.lock.ops:Start node 'smithi118.front.sepia.ceph.com' reimaging
2023-03-19T09:34:41.775 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: reset os type and version on server
2023-03-19T09:34:41.775 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server
2023-03-19T09:34:41.804 INFO:teuthology.lock.ops:Node 'smithi118.front.sepia.ceph.com' reimaging is complete
2023-03-19T09:34:42.577 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging
2023-03-19T09:34:44.128 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server
2023-03-19T09:34:44.129 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2023-03-19T09:34:44.154 INFO:teuthology.lock.ops:Node 'smithi170.front.sepia.ceph.com' reimaging is complete
2023-03-19T09:34:49.524 INFO:teuthology.provision.fog.smithi111:Scheduling deploy of centos 8.stream
2023-03-19T09:34:50.876 INFO:teuthology.provision.fog.smithi118:Scheduling deploy of centos 8.stream
2023-03-19T09:34:52.983 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.stream
2023-03-19T09:34:54.196 INFO:teuthology.orchestra.console:Power off smithi111
2023-03-19T09:34:56.287 INFO:teuthology.orchestra.console:Power off smithi118
2023-03-19T09:34:57.374 INFO:teuthology.orchestra.console:Power off for smithi111 completed
2023-03-19T09:34:59.326 INFO:teuthology.orchestra.console:Power off smithi170
2023-03-19T09:35:03.440 INFO:teuthology.orchestra.console:Power on smithi111
2023-03-19T09:35:04.928 INFO:teuthology.orchestra.console:Power off for smithi118 completed
2023-03-19T09:35:06.343 INFO:teuthology.orchestra.console:Power on smithi118
2023-03-19T09:35:13.071 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2023-03-19T09:35:14.012 INFO:teuthology.orchestra.console:Power on for smithi111 completed
2023-03-19T09:35:14.112 INFO:teuthology.orchestra.console:Power on smithi170
2023-03-19T09:35:15.213 INFO:teuthology.provision.fog.smithi111:Waiting for deploy to finish
2023-03-19T09:35:16.572 INFO:teuthology.orchestra.console:Power on for smithi118 completed
2023-03-19T09:35:17.497 INFO:teuthology.provision.fog.smithi118:Waiting for deploy to finish
2023-03-19T09:35:20.601 INFO:teuthology.orchestra.console:Power on for smithi170 completed
2023-03-19T09:35:35.553 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2023-03-19T09:38:24.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111
2023-03-19T09:38:27.240 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-19T09:38:27.240 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-03-19T09:38:36.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111
2023-03-19T09:38:36.950 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-19T09:38:36.950 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-03-19T09:38:46.021 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111
2023-03-19T09:38:48.883 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-19T09:38:48.883 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170
2023-03-19T09:38:57.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111
2023-03-19T09:39:09.484 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-19T09:39:18.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111
2023-03-19T09:39:19.590 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-19T09:39:28.645 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.111
2023-03-19T09:39:30.469 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.118
2023-03-19T09:39:54.909 WARNING:teuthology.provision.fog:timed out
2023-03-19T09:40:02.878 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-19T09:40:04.517 INFO:teuthology.provision.fog.smithi170:Node is ready
2023-03-19T09:40:06.104 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com
2023-03-19T09:40:07.110 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170
2023-03-19T09:40:11.022 INFO:teuthology.provision.fog.smithi170:Deploy complete!
2023-03-19T09:40:16.511 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-19T09:40:16.511 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-19T09:40:26.557 INFO:teuthology.provision.fog.smithi111:Node is ready
2023-03-19T09:40:27.837 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111.front.sepia.ceph.com
2023-03-19T09:40:31.419 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111 smithi111.front.sepia.ceph.com smithi111
2023-03-19T09:40:32.499 INFO:teuthology.provision.fog.smithi118:Node is ready
2023-03-19T09:40:33.567 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118.front.sepia.ceph.com
2023-03-19T09:40:35.434 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118 smithi118.front.sepia.ceph.com smithi118
2023-03-19T09:40:37.184 INFO:teuthology.provision.fog.smithi111:Deploy complete!
2023-03-19T09:40:41.577 INFO:teuthology.provision.fog.smithi118:Deploy complete!
2023-03-19T09:40:42.686 INFO:teuthology.lock.ops:Checking smithi111.front.sepia.ceph.com
2023-03-19T09:40:43.571 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-19T09:40:44.982 INFO:teuthology.lock.ops:Checking smithi118.front.sepia.ceph.com
2023-03-19T09:40:47.000 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-19T09:40:49.115 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com
2023-03-19T09:40:50.223 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-19T09:40:50.250 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: set os type and version on server
2023-03-19T09:40:52.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T09:40:52.575 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64
2023-03-19T09:40:55.025 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/"
2023-03-19T09:40:56.441 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-19T09:40:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-19T09:40:56.442 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-19T09:40:56.442 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server
2023-03-19T09:40:56.464 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: set os type and version on server
2023-03-19T09:40:59.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T09:41:00.187 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64
2023-03-19T09:41:02.119 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream"
2023-03-19T09:41:04.462 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8"
2023-03-19T09:41:05.191 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos"
2023-03-19T09:41:05.191 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora"
2023-03-19T09:41:05.191 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8"
2023-03-19T09:41:05.191 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8"
2023-03-19T09:41:05.191 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-19T09:41:05.191 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31"
2023-03-19T09:41:05.191 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-19T09:41:05.192 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/"
2023-03-19T09:41:05.192 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-19T09:41:05.192 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-19T09:41:05.192 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-19T09:41:05.192 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server
2023-03-19T09:41:05.213 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server
2023-03-19T09:41:07.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T09:41:08.184 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64
2023-03-19T09:41:09.804 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-19T09:41:11.304 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-19T09:41:11.305 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-19T09:41:11.306 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2023-03-19T09:41:14.449 INFO:teuthology.dispatcher.supervisor:Running job 7212342
2023-03-19T09:41:16.561 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi/7212342 --name yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} -- /home/teuthworker/archive/yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi/7212342/orig.config.yaml
2023-03-19T09:41:17.745 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi/7212342
2023-03-19T09:41:19.903 INFO:teuthology.dispatcher.supervisor:Job PID: 8543
2023-03-19T09:41:19.903 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-19T21:42:12.435 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T21:42:12.888 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:42:16.007 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T21:42:16.543 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T21:42:16.543 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T21:42:16.543 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:42:16.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T21:42:16.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T21:42:18.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T21:42:28.147 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:42:31.252 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-19T21:42:34.771 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-19T21:42:36.812 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:42:50.830 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/8c014f8a-c63d-11ed-9afc-001a4aab830c/ceph-osd.9.log: file size changed while zipping
2023-03-19T21:42:54.430 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/8c014f8a-c63d-11ed-9afc-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-03-19T21:43:06.168 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/8c014f8a-c63d-11ed-9afc-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-19T21:43:12.159 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/8c014f8a-c63d-11ed-9afc-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-19T21:43:22.087 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/8c014f8a-c63d-11ed-9afc-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-19T21:43:29.881 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/8c014f8a-c63d-11ed-9afc-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2023-03-19T21:43:39.433 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /var/log/ceph/8c014f8a-c63d-11ed-9afc-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-19T21:44:05.502 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:44:07.823 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:46:10.353 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T21:46:11.663 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:46:13.047 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:46:13.737 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T21:46:15.621 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T21:46:15.621 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T21:46:15.622 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:46:16.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T21:46:16.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T21:46:17.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T21:46:18.617 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:46:19.666 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:46:45.427 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:46:46.015 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:48:47.179 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T21:48:48.591 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:48:49.562 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:48:51.340 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T21:48:52.831 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T21:48:52.832 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T21:48:52.832 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:48:52.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T21:48:52.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T21:48:53.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T21:48:55.602 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:48:58.341 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:49:24.664 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:49:26.406 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:51:28.117 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T21:51:29.038 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:51:30.909 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:51:33.427 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T21:51:34.681 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T21:51:34.681 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T21:51:34.682 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:51:34.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T21:51:34.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T21:51:35.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T21:51:37.023 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:51:38.185 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:52:05.119 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:52:05.965 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:54:07.812 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T21:54:09.422 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:54:10.600 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:54:12.338 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T21:54:13.368 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T21:54:13.368 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T21:54:13.369 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:54:16.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T21:54:17.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T21:54:18.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T21:54:24.265 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:54:26.903 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:54:54.989 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:54:58.166 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:57:03.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T21:57:04.652 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:57:05.572 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:57:07.492 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T21:57:09.661 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T21:57:11.176 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T21:57:11.845 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:57:11.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T21:57:12.020 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T21:57:13.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T21:57:16.624 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:57:19.355 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:57:47.943 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:57:48.427 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:59:49.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T21:59:50.097 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T21:59:51.073 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T21:59:52.524 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T21:59:57.850 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T21:59:57.851 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T21:59:57.851 INFO:teuthology.misc:Compressing logs...
2023-03-19T21:59:57.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T21:59:58.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T21:59:59.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T22:00:00.033 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:00:01.336 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:00:33.083 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T22:00:37.797 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T22:02:40.145 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T22:02:40.463 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T22:02:41.266 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T22:02:42.619 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T22:02:43.441 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T22:02:43.441 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T22:02:43.441 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:02:44.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T22:02:44.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T22:02:45.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T22:02:46.271 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:02:47.438 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:03:13.194 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T22:03:14.099 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T22:05:15.580 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T22:05:17.859 INFO:teuthology.kill:Killing Pids: {8543}
2023-03-19T22:05:19.388 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T22:05:21.849 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
2023-03-19T22:05:24.090 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
2023-03-19T22:05:24.090 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2']
2023-03-19T22:05:24.091 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:05:24.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com'
2023-03-19T22:05:24.269 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com'
2023-03-19T22:05:25.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2023-03-19T22:05:26.888 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:05:33.528 INFO:teuthology.misc:Compressing logs...
2023-03-19T22:06:01.206 INFO:teuthology.kill:No teuthology processes running
2023-03-19T22:06:02.776 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'}
2023-03-19T22:08:03.010 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-19T22:08:04.309 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-19T22:08:04.790 INFO:teuthology.nuke:Checking targets against current locks
2023-03-19T22:08:05.927 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T22:08:07.538 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T22:08:08.805 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T22:08:09.198 INFO:teuthology.orchestra.console:Power off smithi111
2023-03-19T22:08:10.981 INFO:teuthology.orchestra.console:Power off smithi170
2023-03-19T22:08:12.213 INFO:teuthology.orchestra.console:Power off smithi118
2023-03-19T22:08:21.533 INFO:teuthology.orchestra.console:Power off for smithi111 completed
2023-03-19T22:08:28.059 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2023-03-19T22:08:28.059 INFO:teuthology.orchestra.console:Power off for smithi118 completed
2023-03-19T22:08:28.220 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com
2023-03-19T22:08:37.018 INFO:teuthology.lock.ops:unlocked: smithi111.front.sepia.ceph.com
2023-03-19T22:08:37.018 INFO:teuthology.lock.ops:unlocked: smithi118.front.sepia.ceph.com