2023-03-16T16:48:02.499 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f
2023-03-16T16:48:06.308 INFO:teuthology.lock.ops:Start node 'smithi140.front.sepia.ceph.com' reimaging
2023-03-16T16:48:08.309 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T16:48:09.111 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-03-16T16:48:09.137 INFO:teuthology.lock.ops:Node 'smithi140.front.sepia.ceph.com' reimaging is complete
2023-03-16T16:48:14.113 INFO:teuthology.lock.ops:Start node 'smithi155.front.sepia.ceph.com' reimaging
2023-03-16T16:48:15.198 INFO:teuthology.lock.ops:Updating [smithi155.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T16:48:17.047 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server
2023-03-16T16:48:17.078 INFO:teuthology.lock.ops:Node 'smithi155.front.sepia.ceph.com' reimaging is complete
2023-03-16T16:48:18.615 INFO:teuthology.provision.fog.smithi140:Scheduling deploy of ubuntu 20.04
2023-03-16T16:48:21.719 INFO:teuthology.provision.fog.smithi155:Scheduling deploy of ubuntu 20.04
2023-03-16T16:48:25.369 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-16T16:48:27.117 INFO:teuthology.orchestra.console:Power off smithi155
2023-03-16T16:48:29.401 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-16T16:48:31.382 INFO:teuthology.orchestra.console:Power off for smithi155 completed
2023-03-16T16:48:31.483 INFO:teuthology.orchestra.console:Power on smithi140
2023-03-16T16:48:33.316 INFO:teuthology.orchestra.console:Power on smithi155
2023-03-16T16:48:40.584 INFO:teuthology.orchestra.console:Power on for smithi140 completed
2023-03-16T16:48:42.660 INFO:teuthology.provision.fog.smithi140:Waiting for deploy to finish
2023-03-16T16:48:44.878 INFO:teuthology.orchestra.console:Power on for smithi155 completed
2023-03-16T16:48:45.799 INFO:teuthology.provision.fog.smithi155:Waiting for deploy to finish
2023-03-16T16:52:35.621 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-03-16T16:52:45.829 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-03-16T16:52:55.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-03-16T16:53:04.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-03-16T16:53:05.578 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-16T16:53:14.661 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-03-16T16:53:15.635 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-16T16:53:24.713 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-03-16T16:53:25.942 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-16T16:53:32.485 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.140
2023-03-16T16:53:35.017 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.155
2023-03-16T16:54:39.685 WARNING:teuthology.provision.fog:timed out
2023-03-16T16:54:42.295 WARNING:teuthology.provision.fog:timed out
2023-03-16T16:54:47.356 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T16:54:49.180 INFO:teuthology.provision.fog.smithi140:Node is ready
2023-03-16T16:54:51.040 INFO:teuthology.orchestra.run.smithi140.stdout:smithi140.front.sepia.ceph.com
2023-03-16T16:54:53.548 INFO:teuthology.orchestra.run.smithi140.stdout:172.21.15.140 smithi140.front.sepia.ceph.com smithi140
2023-03-16T16:54:55.376 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T16:54:56.382 INFO:teuthology.provision.fog.smithi140:Deploy complete!
2023-03-16T16:54:56.417 INFO:teuthology.provision.fog.smithi155:Node is ready
2023-03-16T16:54:57.250 INFO:teuthology.orchestra.run.smithi155.stdout:smithi155.front.sepia.ceph.com
2023-03-16T16:54:58.022 INFO:teuthology.orchestra.run.smithi155.stdout:172.21.15.155 smithi155.front.sepia.ceph.com smithi155
2023-03-16T16:55:00.712 INFO:teuthology.provision.fog.smithi155:Deploy complete!
2023-03-16T16:55:01.608 INFO:teuthology.lock.ops:Checking smithi140.front.sepia.ceph.com
2023-03-16T16:55:03.766 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T16:55:03.792 INFO:teuthology.lock.ops:Checking smithi155.front.sepia.ceph.com
2023-03-16T16:55:05.472 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T16:55:05.502 INFO:teuthology.lock.ops:Updating [smithi140.front.sepia.ceph.com]: set os type and version on server
2023-03-16T16:55:06.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-16T16:55:07.252 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64
2023-03-16T16:55:07.863 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="Ubuntu"
2023-03-16T16:55:09.149 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-03-16T16:55:09.149 INFO:teuthology.orchestra.run.smithi140.stdout:ID=ubuntu
2023-03-16T16:55:09.150 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE=debian
2023-03-16T16:55:09.395 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-03-16T16:55:10.509 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="20.04"
2023-03-16T16:55:10.509 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://www.ubuntu.com/"
2023-03-16T16:55:10.509 INFO:teuthology.orchestra.run.smithi140.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-03-16T16:55:10.509 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-03-16T16:55:10.509 INFO:teuthology.orchestra.run.smithi140.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-03-16T16:55:11.433 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_CODENAME=focal
2023-03-16T16:55:11.433 INFO:teuthology.orchestra.run.smithi140.stdout:UBUNTU_CODENAME=focal
2023-03-16T16:55:11.434 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server
2023-03-16T16:55:11.466 INFO:teuthology.lock.ops:Updating [smithi155.front.sepia.ceph.com]: set os type and version on server
2023-03-16T16:55:12.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-16T16:55:12.540 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64
2023-03-16T16:55:14.545 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Ubuntu"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="20.04.4 LTS (Focal Fossa)"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:ID=ubuntu
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE=debian
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="20.04"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.ubuntu.com/"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_CODENAME=focal
2023-03-16T16:55:15.558 INFO:teuthology.orchestra.run.smithi155.stdout:UBUNTU_CODENAME=focal
2023-03-16T16:55:15.559 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server
2023-03-16T16:55:15.683 INFO:teuthology.dispatcher.supervisor:Running job 7207102
2023-03-16T16:55:16.929 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-14_20:10:47-rados-wip-yuri-testing-2023-03-14-0714-reef-distro-default-smithi/7207102 --name yuriw-2023-03-14_20:10:47-rados-wip-yuri-testing-2023-03-14-0714-reef-distro-default-smithi --description rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{ubuntu_latest} tasks/insights} -- /home/teuthworker/archive/yuriw-2023-03-14_20:10:47-rados-wip-yuri-testing-2023-03-14-0714-reef-distro-default-smithi/7207102/orig.config.yaml
2023-03-16T16:55:17.351 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-14_20:10:47-rados-wip-yuri-testing-2023-03-14-0714-reef-distro-default-smithi/7207102
2023-03-16T16:55:18.146 INFO:teuthology.dispatcher.supervisor:Job PID: 7899
2023-03-16T16:55:18.146 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-17T04:56:10.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-17T04:56:11.750 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T04:56:14.251 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
KeyError: 'status'
2023-03-17T04:56:16.213 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0']
2023-03-17T04:56:18.107 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']
2023-03-17T04:56:18.722 INFO:teuthology.misc:Compressing logs...
2023-03-17T04:56:18.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-17T04:56:18.936 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-17T04:56:20.230 INFO:teuthology.misc:Compressing logs...
2023-03-17T04:56:21.502 INFO:teuthology.misc:Compressing logs...
2023-03-17T04:56:30.455 INFO:teuthology.orchestra.run.smithi155.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping
2023-03-17T04:56:31.452 INFO:teuthology.orchestra.run.smithi155.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping
2023-03-17T04:56:38.148 INFO:teuthology.orchestra.run.smithi155.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping
2023-03-17T04:56:43.339 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping
2023-03-17T04:56:46.621 INFO:teuthology.orchestra.run.smithi155.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping
2023-03-17T04:56:47.189 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping
2023-03-17T04:56:53.401 INFO:teuthology.orchestra.run.smithi155.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping
2023-03-17T04:56:56.448 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping
2023-03-17T04:57:03.375 INFO:teuthology.orchestra.run.smithi140.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping
2023-03-17T04:57:30.538 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T04:57:31.728 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T04:59:33.097 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-17T04:59:35.809 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T04:59:37.211 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T04:59:39.067 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0']
2023-03-17T04:59:40.071 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']
2023-03-17T04:59:40.071 INFO:teuthology.misc:Compressing logs...
2023-03-17T04:59:40.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-17T04:59:40.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-17T04:59:41.585 INFO:teuthology.misc:Compressing logs...
2023-03-17T04:59:42.964 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:00:10.434 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:00:15.105 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:02:16.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-17T05:02:17.681 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:02:19.036 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:02:20.896 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0']
2023-03-17T05:02:21.998 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']
2023-03-17T05:02:21.998 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:02:21.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-17T05:02:22.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-17T05:02:23.853 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:02:25.641 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:02:53.975 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:02:54.223 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:04:57.201 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-17T05:04:59.367 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:05:00.773 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:05:03.547 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0']
2023-03-17T05:05:05.496 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']
2023-03-17T05:05:05.496 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:05:06.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-17T05:05:06.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-17T05:05:09.118 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:05:10.136 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:05:38.462 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:05:39.977 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:07:41.349 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-17T05:07:41.390 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:07:41.920 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:07:42.880 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0']
2023-03-17T05:07:44.166 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']
2023-03-17T05:07:44.166 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:07:44.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-17T05:07:44.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-17T05:07:45.002 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:07:46.445 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:08:14.008 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:08:15.947 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:10:17.884 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-17T05:10:19.737 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:10:20.971 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:10:22.934 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0']
2023-03-17T05:10:24.524 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']
2023-03-17T05:10:24.525 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:10:24.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-17T05:10:24.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-17T05:10:26.715 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:10:30.439 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:10:57.416 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:10:59.918 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:13:01.376 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-17T05:13:03.997 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:13:04.756 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:13:05.651 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0']
2023-03-17T05:13:06.065 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1']
2023-03-17T05:13:07.088 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:13:07.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com'
2023-03-17T05:13:07.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com'
2023-03-17T05:13:08.162 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:13:09.488 INFO:teuthology.misc:Compressing logs...
2023-03-17T05:13:38.895 INFO:teuthology.kill:Killing Pids: {7899}
2023-03-17T05:13:40.488 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'}
2023-03-17T05:15:42.165 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-17T05:15:43.505 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-17T05:15:44.229 INFO:teuthology.nuke:Checking targets against current locks
2023-03-17T05:15:44.283 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-17T05:15:46.722 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-17T05:15:48.507 INFO:teuthology.orchestra.console:Power off smithi140
2023-03-17T05:15:50.308 INFO:teuthology.orchestra.console:Power off smithi155
2023-03-17T05:16:00.909 INFO:teuthology.orchestra.console:Power off for smithi140 completed
2023-03-17T05:16:01.845 INFO:teuthology.orchestra.console:Power off for smithi155 completed
2023-03-17T05:16:02.004 INFO:teuthology.lock.ops:unlocked: smithi140.front.sepia.ceph.com
2023-03-17T05:16:02.710 INFO:teuthology.lock.ops:unlocked: smithi155.front.sepia.ceph.com