2023-03-18T16:26:42.900 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-18T16:26:44.719 INFO:teuthology.lock.ops:Start node 'smithi003.front.sepia.ceph.com' reimaging 2023-03-18T16:26:46.479 INFO:teuthology.lock.ops:Updating [smithi003.front.sepia.ceph.com]: reset os type and version on server 2023-03-18T16:26:46.479 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-03-18T16:26:46.502 INFO:teuthology.lock.ops:Node 'smithi003.front.sepia.ceph.com' reimaging is complete 2023-03-18T16:26:48.671 INFO:teuthology.lock.ops:Start node 'smithi042.front.sepia.ceph.com' reimaging 2023-03-18T16:26:48.671 INFO:teuthology.lock.ops:Updating [smithi042.front.sepia.ceph.com]: reset os type and version on server 2023-03-18T16:26:48.671 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2023-03-18T16:26:48.701 INFO:teuthology.lock.ops:Node 'smithi042.front.sepia.ceph.com' reimaging is complete 2023-03-18T16:26:51.209 INFO:teuthology.provision.fog.smithi003:Scheduling deploy of centos 8.stream 2023-03-18T16:26:54.378 INFO:teuthology.provision.fog.smithi042:Scheduling deploy of centos 8.stream 2023-03-18T16:26:56.304 INFO:teuthology.orchestra.console:Power off smithi003 2023-03-18T16:26:57.347 INFO:teuthology.orchestra.console:Power off smithi042 2023-03-18T16:27:09.767 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2023-03-18T16:27:11.059 INFO:teuthology.orchestra.console:Power off for smithi042 completed 2023-03-18T16:27:11.159 INFO:teuthology.orchestra.console:Power on smithi003 2023-03-18T16:27:13.602 INFO:teuthology.orchestra.console:Power on smithi042 2023-03-18T16:27:18.995 INFO:teuthology.orchestra.console:Power on for smithi003 completed 2023-03-18T16:27:22.340 INFO:teuthology.orchestra.console:Power on for smithi042 completed 2023-03-18T16:27:22.441 INFO:teuthology.provision.fog.smithi003:Waiting for deploy to finish 2023-03-18T16:27:23.519 INFO:teuthology.provision.fog.smithi042:Waiting for deploy to finish 2023-03-18T16:29:57.253 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-03-18T16:30:07.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-03-18T16:30:12.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.42 2023-03-18T16:30:18.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-03-18T16:30:24.041 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.42 2023-03-18T16:30:30.053 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-03-18T16:30:35.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.42 2023-03-18T16:30:39.685 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.3 2023-03-18T16:30:45.317 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.42 2023-03-18T16:31:47.389 WARNING:teuthology.provision.fog:timed out 2023-03-18T16:31:52.925 WARNING:teuthology.provision.fog:timed out 2023-03-18T16:31:56.316 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-18T16:31:57.283 INFO:teuthology.provision.fog.smithi003:Node is ready 2023-03-18T16:31:57.672 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003.front.sepia.ceph.com 2023-03-18T16:32:00.799 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3 smithi003.front.sepia.ceph.com smithi003 2023-03-18T16:32:02.240 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-18T16:32:04.053 INFO:teuthology.provision.fog.smithi003:Deploy complete! 2023-03-18T16:32:04.096 INFO:teuthology.provision.fog.smithi042:Node is ready 2023-03-18T16:32:07.085 INFO:teuthology.orchestra.run.smithi042.stdout:smithi042.front.sepia.ceph.com 2023-03-18T16:32:08.768 INFO:teuthology.orchestra.run.smithi042.stdout:172.21.15.42 smithi042.front.sepia.ceph.com smithi042 2023-03-18T16:32:11.331 INFO:teuthology.provision.fog.smithi042:Deploy complete! 2023-03-18T16:32:14.346 INFO:teuthology.lock.ops:Checking smithi003.front.sepia.ceph.com 2023-03-18T16:32:15.602 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-18T16:32:15.633 INFO:teuthology.lock.ops:Checking smithi042.front.sepia.ceph.com 2023-03-18T16:32:17.419 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-18T16:32:17.450 INFO:teuthology.lock.ops:Updating [smithi003.front.sepia.ceph.com]: set os type and version on server 2023-03-18T16:32:19.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-18T16:32:19.387 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2023-03-18T16:32:21.069 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2023-03-18T16:32:22.453 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8" 2023-03-18T16:32:22.453 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2023-03-18T16:32:22.453 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2023-03-18T16:32:22.453 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8" 2023-03-18T16:32:22.454 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2023-03-18T16:32:22.454 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-18T16:32:22.454 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2023-03-18T16:32:24.084 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-18T16:32:24.084 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2023-03-18T16:32:24.084 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-18T16:32:24.084 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-18T16:32:24.084 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-18T16:32:24.085 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2023-03-18T16:32:24.106 INFO:teuthology.lock.ops:Updating [smithi042.front.sepia.ceph.com]: set os type and version on server 2023-03-18T16:32:27.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-18T16:32:27.410 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2023-03-18T16:32:30.149 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="CentOS Stream" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="8" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:ID="centos" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE="rhel fedora" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="8" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:PLATFORM_ID="platform:el8" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:ANSI_COLOR="0;31" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://centos.org/" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-18T16:32:31.855 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-18T16:32:33.131 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-18T16:32:33.131 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2023-03-18T16:32:33.242 INFO:teuthology.dispatcher.supervisor:Running job 7209695 2023-03-18T16:32:35.128 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209695 --name yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi --description rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{centos_8} tasks/failover} -- /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209695/orig.config.yaml 2023-03-18T16:32:35.132 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209695 2023-03-18T16:32:36.586 INFO:teuthology.dispatcher.supervisor:Job PID: 10865 2023-03-18T16:32:36.586 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-03-19T04:33:27.845 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:33:30.934 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:33:36.047 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) KeyError: 'status' 2023-03-19T04:33:37.496 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:33:38.774 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:33:38.774 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:33:38.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:33:38.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:33:39.963 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:33:40.455 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:33:58.537 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-03-19T04:34:03.629 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-03-19T04:34:16.123 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-03-19T04:34:16.956 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-03-19T04:34:20.651 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-03-19T04:34:29.842 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-03-19T04:34:33.121 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-03-19T04:34:38.158 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-03-19T04:34:47.836 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-03-19T04:34:48.517 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log: file size changed while zipping 2023-03-19T04:35:12.422 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:35:16.339 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:37:17.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:37:20.049 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:37:23.082 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:37:25.470 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:37:27.815 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:37:29.613 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:37:29.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:37:29.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:37:31.267 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:37:32.583 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:37:57.439 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:37:58.467 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:40:00.049 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:40:01.523 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:40:03.499 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:40:06.600 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:40:08.601 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:40:08.601 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:40:08.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:40:08.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:40:11.114 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:40:11.926 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:40:37.896 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:40:38.821 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:42:40.650 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:42:41.336 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:42:42.117 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:42:42.949 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:42:43.952 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:42:43.952 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:42:43.952 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:42:44.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:42:45.300 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:42:46.692 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:43:12.134 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:43:13.375 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:45:15.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:45:17.228 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:45:21.175 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:45:23.051 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:45:24.201 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:45:24.202 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:45:26.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:45:26.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:45:27.923 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:45:29.990 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:45:54.885 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:45:55.492 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:47:56.970 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:47:58.138 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:47:59.008 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:48:00.363 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:48:02.275 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:48:02.275 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:48:02.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:48:02.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:48:05.476 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:48:06.594 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:48:34.011 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:48:37.219 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:50:38.969 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:50:40.322 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:50:43.551 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:50:44.762 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:50:46.057 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:50:46.057 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:50:46.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:50:46.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:50:48.071 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:50:49.492 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:51:14.309 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:51:15.429 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:53:16.269 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:53:18.189 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:53:19.948 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:53:23.339 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:53:24.656 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:53:24.657 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:53:24.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:53:26.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:53:29.478 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:53:30.293 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:53:55.581 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:53:58.266 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:56:01.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:56:03.125 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:56:03.937 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:56:04.954 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:56:05.863 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:56:05.863 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:56:05.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:56:06.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:56:07.110 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:56:08.403 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:56:33.693 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:56:34.532 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:58:36.797 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T04:58:37.667 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:58:39.197 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T04:58:41.386 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T04:58:43.222 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T04:58:43.223 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:58:43.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T04:58:43.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T04:58:44.764 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:58:46.077 INFO:teuthology.misc:Compressing logs... 2023-03-19T04:59:10.949 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T04:59:12.968 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:01:14.965 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:01:15.650 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:01:17.305 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:01:18.719 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:01:20.302 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:01:20.302 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:01:20.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:01:20.494 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:01:21.892 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:01:23.032 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:01:49.588 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:01:50.780 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:03:52.529 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:03:54.492 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:03:56.357 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:03:58.290 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:03:59.088 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:03:59.088 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:03:59.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:04:01.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:04:04.184 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:04:05.172 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:04:06.641 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:04:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:04:08.415 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:04:10.135 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:04:10.176 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:04:12.820 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:06:14.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:06:15.960 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:06:16.530 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:06:18.177 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:06:19.860 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:06:19.860 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:06:19.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:06:20.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:06:21.692 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:06:24.019 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:06:27.167 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:06:29.183 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:06:29.183 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:06:29.183 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:06:29.229 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:06:32.699 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:08:33.456 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:08:33.660 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:08:34.922 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:08:36.174 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:08:37.390 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:08:41.389 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:08:41.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:08:41.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:08:44.219 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:08:48.389 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:08:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:08:50.937 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:08:50.937 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:08:50.937 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:08:50.980 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:08:52.126 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:10:53.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:10:55.586 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:10:57.224 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:10:58.174 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:11:00.081 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:11:00.081 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:11:00.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:11:00.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:11:01.747 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:11:04.118 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:11:07.460 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:11:11.178 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:11:11.178 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:11:11.178 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:11:11.215 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:11:13.012 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:13:14.817 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:13:16.160 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:13:17.188 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:13:18.383 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:13:20.514 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:13:20.515 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:13:23.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:13:23.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:13:25.417 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:13:27.126 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:13:28.412 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:13:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:13:30.299 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:13:30.300 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:13:30.339 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:13:31.316 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:15:32.643 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:15:34.493 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:15:37.736 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:15:43.353 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:15:44.396 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:15:44.396 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:15:44.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:15:44.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:15:46.314 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:15:47.024 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:15:47.697 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:15:48.893 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:15:49.732 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:15:49.732 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:15:50.989 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:15:52.136 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:17:52.964 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:17:55.139 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:17:57.557 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:17:59.687 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:18:01.861 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:18:01.861 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:18:01.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:18:02.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:18:04.094 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:18:05.898 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:18:09.010 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:18:10.880 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:18:12.882 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:18:12.882 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:18:14.179 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:18:17.993 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:20:19.929 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:20:21.746 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:20:22.946 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:20:24.361 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:20:25.185 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:20:25.185 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:20:25.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:20:25.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:20:27.087 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:20:27.389 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:20:28.806 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:20:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:20:30.571 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:20:31.706 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:20:31.743 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:20:32.751 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:22:33.733 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:22:34.673 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:22:35.994 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:22:36.942 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:22:39.215 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:22:39.215 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:22:39.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:22:40.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:22:42.005 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:22:43.689 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:22:44.702 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:22:46.387 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:22:46.387 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:22:47.878 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:22:47.947 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:22:50.377 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:24:53.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:24:55.679 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:24:59.324 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:25:01.187 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:25:03.437 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:25:03.437 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:25:04.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:25:04.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:25:06.003 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:25:06.792 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:25:07.495 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:25:10.331 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:25:10.332 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:25:10.332 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:25:10.368 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:25:11.520 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:27:13.150 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:27:16.011 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:27:17.197 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:27:18.272 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:27:19.485 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:27:20.916 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:27:20.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:27:21.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:27:22.741 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:27:23.768 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:27:24.897 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:27:27.305 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:27:28.851 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:27:28.852 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:27:28.890 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:27:31.069 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:29:36.175 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:29:37.934 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:29:40.155 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:29:42.270 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:29:44.071 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:29:44.071 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:29:44.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:29:44.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:29:47.073 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:29:50.199 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:29:51.912 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:29:53.252 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:29:53.252 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:29:53.252 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:29:53.288 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:29:54.534 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:31:57.888 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:31:59.219 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:32:00.759 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:32:02.483 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:32:03.483 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:32:04.416 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:32:04.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:32:04.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:32:05.826 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:32:07.146 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:32:09.366 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:32:10.527 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:32:10.527 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:32:10.528 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:32:10.564 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:32:12.968 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:34:14.998 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:34:15.762 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:34:17.468 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:34:18.935 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:34:21.373 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:34:21.374 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:34:22.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:34:22.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:34:24.550 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:34:26.224 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:34:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:34:31.185 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:34:33.372 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:34:33.372 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:34:33.408 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:34:35.434 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:36:37.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:36:38.363 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:36:40.913 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:36:43.851 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:36:45.119 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:36:46.279 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:36:46.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:36:46.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:36:47.871 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:36:48.846 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:36:52.526 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:36:54.195 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:36:54.195 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:36:54.195 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:36:54.233 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:36:58.667 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:39:03.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T05:39:04.144 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:39:05.659 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:39:07.881 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-19T05:39:09.734 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-19T05:39:09.734 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:39:09.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2023-03-19T05:39:09.921 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2023-03-19T05:39:11.620 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:39:13.140 INFO:teuthology.misc:Compressing logs... 2023-03-19T05:39:14.217 INFO:teuthology.orchestra.run.smithi003.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-19T05:39:14.806 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-19T05:39:15.305 INFO:teuthology.orchestra.run.smithi042.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-19T05:39:15.305 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi003 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T05:39:15.345 INFO:teuthology.kill:Killing Pids: {10865} 2023-03-19T05:39:16.089 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'} 2023-03-19T05:41:17.833 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-03-19T05:41:19.559 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-03-19T05:41:23.158 INFO:teuthology.nuke:Checking targets against current locks 2023-03-19T05:41:23.218 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-19T05:41:25.688 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-19T05:41:25.762 INFO:teuthology.orchestra.console:Power off smithi042 2023-03-19T05:41:27.050 INFO:teuthology.orchestra.console:Power off smithi003 2023-03-19T05:41:37.501 INFO:teuthology.orchestra.console:Power off for smithi042 completed 2023-03-19T05:41:39.629 INFO:teuthology.lock.ops:unlocked: smithi042.front.sepia.ceph.com 2023-03-19T05:41:41.621 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2023-03-19T05:41:42.263 INFO:teuthology.lock.ops:unlocked: smithi003.front.sepia.ceph.com