2023-03-20T17:35:47.777 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-20T17:35:49.281 INFO:teuthology.lock.ops:Start node 'smithi052.front.sepia.ceph.com' reimaging 2023-03-20T17:35:50.311 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: reset os type and version on server 2023-03-20T17:35:50.311 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2023-03-20T17:35:50.342 INFO:teuthology.lock.ops:Node 'smithi052.front.sepia.ceph.com' reimaging is complete 2023-03-20T17:35:51.285 INFO:teuthology.lock.ops:Start node 'smithi129.front.sepia.ceph.com' reimaging 2023-03-20T17:35:52.572 INFO:teuthology.lock.ops:Updating [smithi129.front.sepia.ceph.com]: reset os type and version on server 2023-03-20T17:35:52.572 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2023-03-20T17:35:52.600 INFO:teuthology.lock.ops:Node 'smithi129.front.sepia.ceph.com' reimaging is complete 2023-03-20T17:35:55.655 INFO:teuthology.provision.fog.smithi052:Scheduling deploy of rhel 8.6 2023-03-20T17:35:59.791 INFO:teuthology.provision.fog.smithi129:Scheduling deploy of rhel 8.6 2023-03-20T17:36:02.183 INFO:teuthology.orchestra.console:Power off smithi052 2023-03-20T17:36:04.664 INFO:teuthology.orchestra.console:Power off smithi129 2023-03-20T17:36:07.122 INFO:teuthology.orchestra.console:Power off for smithi052 completed 2023-03-20T17:36:10.764 INFO:teuthology.orchestra.console:Power off for smithi129 completed 2023-03-20T17:36:10.864 INFO:teuthology.orchestra.console:Power on smithi052 2023-03-20T17:36:12.488 INFO:teuthology.orchestra.console:Power on smithi129 2023-03-20T17:36:19.768 INFO:teuthology.orchestra.console:Power on for smithi052 completed 2023-03-20T17:36:21.324 INFO:teuthology.orchestra.console:Power on for smithi129 completed 2023-03-20T17:36:21.425 INFO:teuthology.provision.fog.smithi052:Waiting for deploy to finish 2023-03-20T17:36:22.502 INFO:teuthology.provision.fog.smithi129:Waiting for deploy to finish 2023-03-20T17:38:56.357 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52 2023-03-20T17:39:06.789 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52 2023-03-20T17:39:13.797 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-03-20T17:39:19.717 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52 2023-03-20T17:39:24.005 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-03-20T17:39:29.989 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.52 2023-03-20T17:39:36.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-03-20T17:39:47.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-03-20T17:40:01.225 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-03-20T17:40:27.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-03-20T17:40:37.829 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-03-20T17:40:39.481 WARNING:teuthology.provision.fog:timed out 2023-03-20T17:40:45.820 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-20T17:41:01.494 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-20T17:41:01.579 INFO:teuthology.provision.fog.smithi052:Node is ready 2023-03-20T17:41:04.209 INFO:teuthology.provision.fog.smithi129:Node is ready 2023-03-20T17:41:05.700 INFO:teuthology.orchestra.run.smithi052.stdout:smithi052.front.sepia.ceph.com 2023-03-20T17:41:05.718 INFO:teuthology.orchestra.run.smithi129.stdout:smithi129.front.sepia.ceph.com 2023-03-20T17:41:07.717 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129 smithi129.front.sepia.ceph.com smithi129 2023-03-20T17:41:09.654 INFO:teuthology.orchestra.run.smithi052.stdout:172.21.15.52 smithi052.front.sepia.ceph.com smithi052 2023-03-20T17:41:10.012 INFO:teuthology.provision.fog.smithi129:Deploy complete! 2023-03-20T17:41:11.140 INFO:teuthology.provision.fog.smithi052:Deploy complete! 2023-03-20T17:41:11.331 INFO:teuthology.lock.ops:Checking smithi052.front.sepia.ceph.com 2023-03-20T17:41:13.097 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-20T17:41:13.118 INFO:teuthology.lock.ops:Checking smithi129.front.sepia.ceph.com 2023-03-20T17:41:14.768 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-20T17:41:14.794 INFO:teuthology.lock.ops:Updating [smithi052.front.sepia.ceph.com]: set os type and version on server 2023-03-20T17:41:15.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-20T17:41:15.335 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2023-03-20T17:41:16.409 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Red Hat Enterprise Linux" 2023-03-20T17:41:19.107 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="8.6 (Ootpa)" 2023-03-20T17:41:19.107 INFO:teuthology.orchestra.run.smithi052.stdout:ID="rhel" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE="fedora" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="8.6" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:PLATFORM_ID="platform:el8" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:ANSI_COLOR="0;31" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.redhat.com/" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-20T17:41:19.108 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-20T17:41:19.109 INFO:teuthology.orchestra.run.smithi052.stdout: 2023-03-20T17:41:19.109 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-20T17:41:19.109 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-20T17:41:19.109 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-20T17:41:19.109 INFO:teuthology.orchestra.run.smithi052.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-20T17:41:19.110 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2023-03-20T17:41:19.134 INFO:teuthology.lock.ops:Updating [smithi129.front.sepia.ceph.com]: set os type and version on server 2023-03-20T17:41:20.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-20T17:41:20.222 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2023-03-20T17:41:22.359 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="Red Hat Enterprise Linux" 2023-03-20T17:41:23.607 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="8.6 (Ootpa)" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:ID="rhel" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="fedora" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="8.6" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el8" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://www.redhat.com/" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout: 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-20T17:41:23.608 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-20T17:41:23.609 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-20T17:41:23.609 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-20T17:41:23.609 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2023-03-20T17:41:23.719 INFO:teuthology.dispatcher.supervisor:Running job 7213113 2023-03-20T17:41:25.625 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213113 --name yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi --description rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{rhel_8} tasks/failover} -- /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213113/orig.config.yaml 2023-03-20T17:41:25.629 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-18_00:57:11-rados-wip-yuri3-testing-2023-03-17-1235-quincy-distro-default-smithi/7213113 2023-03-20T17:41:27.852 INFO:teuthology.dispatcher.supervisor:Job PID: 22971 2023-03-20T17:41:27.852 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-03-21T05:42:20.709 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T05:42:23.240 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:42:27.511 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T05:42:28.510 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T05:42:28.511 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:42:28.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T05:42:28.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T05:42:30.412 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:42:32.515 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:42:51.183 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-03-21T05:42:54.371 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-03-21T05:42:57.543 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log: file size changed while zipping 2023-03-21T05:42:57.544 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-03-21T05:43:04.190 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-03-21T05:43:12.175 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-03-21T05:43:26.707 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-03-21T05:43:29.289 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-03-21T05:43:39.444 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-03-21T05:43:40.370 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-03-21T05:44:03.112 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:44:04.120 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:46:07.081 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T05:46:10.199 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:46:11.439 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:46:12.629 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T05:46:14.152 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T05:46:14.152 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:46:15.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T05:46:15.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T05:46:16.482 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:46:18.415 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:46:48.402 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:46:49.590 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:48:53.658 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T05:48:55.165 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:48:57.490 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:48:59.855 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T05:49:01.804 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T05:49:01.804 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:49:01.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T05:49:02.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T05:49:04.327 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:49:04.971 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:49:30.202 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:49:32.921 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:51:36.356 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T05:51:38.327 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:51:40.733 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:51:42.142 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T05:51:43.498 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T05:51:43.498 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:51:43.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T05:51:43.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T05:51:47.554 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:51:48.593 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:52:14.529 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:52:15.868 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:54:17.098 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T05:54:17.568 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:54:18.865 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:54:19.891 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T05:54:22.073 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T05:54:22.073 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:54:22.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T05:54:22.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T05:54:23.891 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:54:25.188 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:54:49.587 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:54:51.649 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:56:53.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T05:56:54.657 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:56:56.372 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:56:58.571 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T05:57:00.028 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T05:57:00.028 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:57:00.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T05:57:00.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T05:57:01.150 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:57:02.812 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:57:28.439 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:57:30.036 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:59:36.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T05:59:39.369 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T05:59:42.329 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T05:59:43.777 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T05:59:45.668 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T05:59:45.668 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:59:45.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T05:59:45.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T05:59:47.055 INFO:teuthology.misc:Compressing logs... 2023-03-21T05:59:49.069 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:00:14.325 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:00:15.383 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:02:17.221 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:02:18.996 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:02:21.010 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:02:23.645 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:02:25.469 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:02:28.292 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:02:28.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:02:28.478 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:02:29.859 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:02:30.436 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:02:53.966 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:02:54.252 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:04:55.780 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:04:56.732 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:04:57.366 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:04:59.475 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:05:00.616 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:05:00.616 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:05:00.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:05:00.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:05:02.718 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:05:03.651 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:05:28.539 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:05:29.673 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:07:30.841 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:07:32.957 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:07:34.304 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:07:35.736 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:07:36.871 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:07:38.522 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:07:38.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:07:38.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:07:43.140 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:07:44.814 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:08:09.866 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:08:11.522 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:10:13.144 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:10:13.767 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:10:15.210 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:10:16.192 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:10:17.900 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:10:17.900 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:10:17.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:10:18.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:10:19.236 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:10:20.702 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:10:45.256 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:10:46.717 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:12:49.221 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:12:50.870 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:12:52.121 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:12:53.196 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:12:54.472 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:12:54.472 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:12:54.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:12:54.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:12:55.577 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:12:56.519 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:13:21.770 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:13:23.100 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:15:24.768 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:15:28.344 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:15:30.990 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:15:35.976 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:15:38.619 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:15:39.951 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:15:39.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:15:40.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:15:41.384 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:15:42.549 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:16:06.565 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:16:15.888 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:18:18.213 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:18:19.639 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:18:20.358 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:18:21.264 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:18:29.139 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:18:29.139 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:18:29.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:18:29.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:18:31.884 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:18:33.523 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:18:58.457 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:18:59.081 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:21:00.297 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:21:01.134 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:21:02.055 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:21:03.427 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:21:04.708 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:21:04.708 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:21:04.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:21:04.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:21:06.344 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:21:08.455 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:21:35.478 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:21:37.362 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:23:38.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:23:40.317 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:23:41.721 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:23:46.606 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:23:48.762 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:23:48.762 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:23:48.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:23:48.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:23:52.455 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:23:53.228 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:24:19.052 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:24:20.320 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:26:21.758 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:26:22.608 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:26:25.698 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:26:27.681 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:26:30.933 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:26:30.933 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:26:30.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:26:31.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:26:33.560 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:26:35.205 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:27:01.407 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:27:03.067 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:29:05.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:29:05.499 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:29:08.132 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:29:08.755 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:29:09.907 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:29:09.907 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:29:09.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:29:10.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:29:12.085 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:29:15.982 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:29:40.553 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:29:45.586 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:31:46.280 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:31:47.776 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:31:48.696 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:31:50.393 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:31:51.759 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:31:51.759 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:31:51.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:31:51.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:31:53.729 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:31:55.451 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:32:21.625 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:32:22.602 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:34:24.515 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:34:32.470 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:34:34.426 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:34:35.767 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:34:37.442 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:34:37.443 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:34:37.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:34:37.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:34:39.360 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:34:41.458 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:35:06.635 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:35:11.038 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:37:13.461 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:37:19.684 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:37:21.833 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:37:36.876 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:37:38.670 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:37:38.670 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:37:38.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:37:38.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:37:40.169 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:37:42.434 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:38:09.982 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:38:11.172 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:40:12.368 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:40:13.629 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:40:15.621 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:40:17.198 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:40:18.847 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:40:18.847 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:40:18.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:40:19.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:40:20.841 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:40:22.800 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:40:47.622 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:40:56.287 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:42:57.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:42:58.822 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:43:00.216 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:43:01.225 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:43:10.676 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:43:10.676 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:43:10.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:43:10.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:43:14.224 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:43:16.661 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:43:42.135 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:43:43.640 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:45:46.129 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:45:48.248 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:45:49.893 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:45:51.524 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:45:55.356 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:45:55.357 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:45:55.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:45:55.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:45:56.618 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:45:57.916 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:46:24.053 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:46:25.736 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:48:26.834 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:48:29.587 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:48:31.776 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:48:36.320 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:48:41.030 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:48:41.030 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:48:41.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:48:44.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:48:46.769 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:48:47.915 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:48:49.058 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T06:48:50.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T06:48:52.006 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:48:54.601 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:50:54.771 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:50:56.631 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:50:58.366 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:51:01.056 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:51:01.056 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:51:01.056 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:51:01.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:51:01.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:51:02.462 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:51:05.496 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:51:07.918 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T06:51:09.306 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T06:51:12.186 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:51:13.794 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:53:15.744 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:53:16.730 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:53:18.078 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:53:19.685 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:53:20.861 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:53:20.861 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:53:20.861 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:53:21.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:53:22.525 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:53:24.320 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:53:25.943 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T06:53:26.999 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T06:53:27.402 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:53:28.276 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:55:29.953 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:55:31.653 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:55:33.705 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:55:35.326 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:55:37.588 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:55:37.589 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:55:37.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:55:37.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:55:39.574 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:55:41.489 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:55:42.196 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T06:55:43.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T06:55:43.406 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:55:44.493 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:57:45.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T06:57:46.657 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:57:47.950 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T06:57:49.534 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T06:57:50.712 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T06:57:50.712 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:57:51.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T06:57:52.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T06:57:53.390 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:57:54.774 INFO:teuthology.misc:Compressing logs... 2023-03-21T06:57:56.965 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T06:57:58.114 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T06:57:59.895 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T06:58:01.920 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:00:03.945 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:00:05.144 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:00:06.674 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:00:07.631 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:00:09.777 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:00:09.777 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:00:09.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:00:09.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:00:11.861 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:00:14.250 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:00:15.754 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:00:18.385 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:00:18.424 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:00:29.452 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:02:30.925 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:02:31.799 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:02:33.366 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:02:33.765 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:02:34.540 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:02:34.540 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:02:34.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:02:35.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:02:36.359 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:02:36.919 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:02:39.256 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:02:39.888 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:02:39.926 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:02:40.848 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:04:41.823 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:04:51.161 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:04:53.437 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:04:54.920 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:04:55.978 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:04:56.993 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:04:56.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:04:57.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:04:58.253 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:04:59.751 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:05:00.434 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:05:01.323 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:05:01.361 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:05:03.146 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:07:04.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:07:06.586 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:07:08.615 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:07:11.825 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:07:14.508 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:07:14.508 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:07:14.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:07:14.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:07:16.287 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:07:17.532 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:07:19.244 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:07:22.181 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:07:22.220 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:07:24.053 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:09:26.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:09:26.711 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:09:28.679 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:09:29.915 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:09:31.461 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:09:31.462 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:09:31.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:09:31.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:09:35.132 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:09:36.839 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:09:37.726 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:09:38.709 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:09:38.747 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:09:40.150 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:11:41.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:11:41.739 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:11:42.622 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:11:42.667 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:11:43.277 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:11:43.278 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:11:43.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:11:43.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:11:44.408 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:11:44.784 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:11:45.564 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:11:49.071 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:11:50.068 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:11:52.296 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:13:53.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:13:54.941 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:13:56.512 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:13:58.258 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:13:59.901 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:13:59.901 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:13:59.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:14:00.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:14:02.784 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:14:05.515 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:14:07.797 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:14:10.284 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:14:10.322 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:14:12.205 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:16:13.913 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:16:15.288 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:16:16.883 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:16:18.089 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:16:18.280 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:16:18.281 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:16:18.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:16:18.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:16:19.627 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:16:20.724 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:16:22.433 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:16:23.579 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:16:23.616 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:16:24.506 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:18:28.113 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:18:33.110 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:18:34.203 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:18:35.454 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:18:54.019 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:18:54.019 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:18:54.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:18:54.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:18:55.878 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:18:57.677 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:19:06.303 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:19:07.458 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:19:07.497 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:19:08.295 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:21:09.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:21:10.471 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:21:11.196 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:21:11.574 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:21:12.114 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:21:12.114 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:21:12.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:21:12.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:21:13.428 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:21:13.900 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:21:14.257 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:21:14.743 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:21:14.787 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:21:15.707 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:23:17.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:23:23.361 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:23:25.533 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:23:26.384 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:23:28.214 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:23:28.214 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:23:28.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:23:28.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:23:30.077 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:23:31.470 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:23:32.710 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:23:33.566 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:23:33.605 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:23:35.238 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:25:36.193 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:25:37.595 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:25:39.858 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:25:41.899 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:25:44.023 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:25:44.023 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:25:47.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:25:47.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:25:49.736 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:26:05.117 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:26:07.091 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:26:08.178 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:26:09.594 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:26:09.595 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:26:09.636 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:26:11.713 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:28:13.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:28:14.607 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:28:16.774 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:28:18.387 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:28:19.587 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:28:19.587 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:28:19.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:28:19.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:28:22.453 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:28:25.680 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:28:27.214 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:28:29.833 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:28:31.510 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:28:31.510 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:28:31.549 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:28:33.203 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:30:36.327 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:30:36.974 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:30:43.752 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:30:46.432 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:30:47.427 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:30:47.427 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:30:47.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:30:47.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:30:49.475 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:30:50.276 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:30:51.653 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:30:52.541 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:30:52.541 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:30:52.542 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:30:52.588 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:30:53.320 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:32:54.325 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:32:56.291 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:32:58.101 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:32:58.120 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:32:59.753 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:32:59.753 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:32:59.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:32:59.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:33:01.555 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:33:02.767 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:33:04.481 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:33:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:33:05.905 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:33:05.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:33:05.946 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:33:06.934 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:35:08.465 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:35:08.938 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:35:09.822 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:35:18.624 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:35:19.777 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:35:19.986 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:35:19.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:35:20.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:35:20.898 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:35:31.283 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:35:32.980 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:35:33.901 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:35:34.415 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:35:46.442 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:35:46.493 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:35:47.609 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:37:49.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:37:50.984 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:37:52.371 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:37:53.631 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:37:55.722 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:37:55.722 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:37:55.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:37:55.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:37:58.062 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:37:59.962 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:38:01.006 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:38:02.941 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:38:02.941 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:38:02.941 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:38:05.764 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:38:07.186 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:40:10.057 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:40:17.715 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:40:19.198 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:40:20.745 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:40:21.633 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:40:21.633 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:40:21.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:40:21.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:40:22.916 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:40:24.183 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:40:25.080 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:40:34.691 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:40:36.873 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:40:36.873 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:40:38.090 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:40:38.995 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:42:44.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:42:48.176 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:42:50.552 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:42:52.989 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:42:54.535 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:42:54.535 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:42:54.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:42:54.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:42:57.186 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:42:58.583 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:43:01.710 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:43:03.210 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:43:04.491 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:43:04.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:43:04.528 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:43:05.597 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:45:06.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:45:18.794 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:45:20.397 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:45:21.820 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:45:23.390 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:45:23.390 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:45:23.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:45:23.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:45:25.337 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:45:27.717 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:45:29.132 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:45:30.638 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:45:30.638 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:45:30.638 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:45:32.585 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:45:34.865 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:47:37.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:47:38.036 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:47:39.801 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:47:41.657 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:47:43.477 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:47:43.477 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:47:43.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:47:43.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:47:45.045 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:47:46.093 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:47:47.551 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:47:48.339 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:47:48.339 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:47:48.914 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:47:48.952 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:47:49.617 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:49:49.785 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:49:50.712 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:49:52.997 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:49:54.567 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:49:55.519 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:49:55.519 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:49:55.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:49:55.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:49:56.656 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:49:57.566 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:49:58.298 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:49:59.200 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:49:59.200 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:49:59.200 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:49:59.240 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:50:00.267 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:52:01.601 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:52:05.257 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:52:08.095 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:52:10.200 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:52:12.333 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:52:12.333 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:52:12.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:52:14.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:52:17.042 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:52:19.313 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:52:20.944 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:52:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:52:22.416 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:52:22.417 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:52:22.455 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:52:23.188 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:54:24.891 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:54:26.292 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:54:27.469 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:54:28.963 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:54:29.686 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:54:29.687 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:54:29.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:54:29.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:54:31.190 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:54:32.521 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:54:34.100 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:54:35.420 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:54:38.015 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:54:38.015 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:54:38.052 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:54:39.395 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:56:41.053 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:56:43.552 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:56:44.488 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:56:45.348 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:56:46.161 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:56:46.161 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:56:46.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:56:46.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:56:47.380 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:56:47.826 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:56:47.982 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:56:48.629 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:56:48.631 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:56:49.657 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:56:49.698 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:56:50.946 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:58:52.949 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T07:58:54.584 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:58:56.465 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T07:58:59.420 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T07:59:01.684 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T07:59:01.684 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:59:01.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T07:59:01.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T07:59:03.545 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:59:19.084 INFO:teuthology.misc:Compressing logs... 2023-03-21T07:59:21.291 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T07:59:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T07:59:22.815 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T07:59:22.815 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T07:59:22.853 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T07:59:23.457 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:01:24.883 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:01:26.131 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:01:27.280 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:01:28.787 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:01:30.232 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:01:30.232 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:01:30.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:01:30.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:01:31.638 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:01:32.749 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:01:33.814 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:01:38.250 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:01:38.250 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:01:38.250 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:01:38.290 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:01:39.891 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:03:41.880 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:03:43.418 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:03:44.671 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:03:46.140 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:03:47.467 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:03:47.467 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:03:47.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:03:47.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:03:49.402 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:03:50.885 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:03:56.119 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:03:57.931 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:04:00.348 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:04:00.348 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:04:00.390 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:04:01.976 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:06:04.173 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:06:05.556 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:06:07.981 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:06:09.194 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:06:10.983 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:06:10.983 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:06:10.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:06:11.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:06:12.020 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:06:13.345 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:06:14.596 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:06:15.574 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:06:15.574 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:06:15.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:06:15.615 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:06:16.208 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:08:17.329 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:08:18.240 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:08:18.886 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:08:19.604 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:08:20.218 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:08:20.219 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:08:20.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:08:20.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:08:21.182 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:08:22.303 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:08:23.590 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:08:24.450 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:08:27.112 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:08:27.113 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:08:27.153 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:08:29.062 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:10:30.025 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:10:31.688 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:10:33.262 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:10:36.902 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:10:40.151 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:10:40.151 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:10:40.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:10:40.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:10:41.857 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:10:44.634 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:10:47.572 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:10:49.777 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:10:49.777 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:10:49.778 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:10:49.822 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:10:51.838 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:12:53.590 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:12:55.606 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:12:57.865 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:12:59.024 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:13:00.863 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:13:00.863 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:13:02.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:13:02.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:13:03.980 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:13:04.657 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:13:06.372 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:13:07.328 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:13:07.328 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:13:07.328 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:13:07.371 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:13:09.717 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:15:10.881 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:15:14.278 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:15:16.789 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:15:22.234 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:15:23.800 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:15:23.800 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:15:23.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:15:23.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:15:25.355 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:15:27.507 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:15:28.514 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:15:30.336 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:15:30.340 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:15:30.340 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:15:30.379 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:15:31.637 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:17:33.111 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:17:38.098 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:17:40.254 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:17:42.041 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:17:44.088 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:17:44.088 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:17:44.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:17:44.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:17:46.229 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:17:47.541 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:17:49.631 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:17:50.386 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:17:51.644 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:17:51.645 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:17:51.682 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:17:53.125 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:19:53.297 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:19:53.861 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:19:55.493 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:19:56.645 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:19:58.132 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:19:58.133 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:20:00.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:20:00.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:20:02.791 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:20:04.130 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:20:05.341 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:20:07.293 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:20:07.295 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:20:07.866 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:20:09.716 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:20:12.158 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:22:15.241 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:22:16.789 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:22:23.630 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:22:25.134 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:22:31.161 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:22:31.779 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:22:34.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:22:34.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:22:36.987 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:22:39.398 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:22:41.643 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:22:44.428 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:22:45.978 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:22:45.978 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:22:46.037 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:22:47.896 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:24:49.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:24:49.964 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:24:50.874 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:24:52.557 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:24:56.367 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:24:56.367 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:24:56.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:24:56.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:24:59.179 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:25:01.718 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:25:03.864 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:25:05.882 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:25:05.883 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:25:05.883 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:25:05.922 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:25:07.806 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:27:10.296 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:27:11.437 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:27:12.627 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:27:13.248 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:27:14.953 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:27:14.953 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:27:14.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:27:15.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:27:17.194 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:27:18.415 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:27:19.492 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:27:20.245 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:27:20.245 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:27:20.245 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:27:24.162 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:27:25.521 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:29:26.606 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:29:27.212 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:29:30.253 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:29:32.522 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:29:38.433 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:29:38.433 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:29:38.433 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:29:38.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:29:39.920 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:29:40.451 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:29:41.555 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:29:42.510 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:29:42.510 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:29:42.511 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:29:42.556 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:29:43.393 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:31:44.357 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:31:44.856 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:31:45.761 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:31:46.138 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:31:46.465 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:31:46.465 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:31:46.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:31:46.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:31:47.544 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:31:48.165 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:31:49.551 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:31:50.900 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:31:50.901 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:31:50.901 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:31:50.938 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:31:52.187 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:33:53.925 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:33:58.244 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:34:00.528 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:34:00.546 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:34:01.871 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:34:01.871 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:34:01.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:34:02.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:34:11.084 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:34:12.402 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:34:14.270 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:34:15.086 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:34:16.258 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:34:16.259 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:34:16.300 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:34:19.704 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:36:21.569 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:36:22.462 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:36:24.035 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:36:26.265 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:36:37.439 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:36:37.439 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:36:37.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:36:37.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:36:40.420 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:36:42.707 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:36:44.914 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:36:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:36:46.271 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:36:46.271 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:36:46.310 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:36:51.802 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:38:55.413 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:38:58.815 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:39:02.651 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:39:05.733 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:39:07.580 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:39:07.580 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:39:07.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:39:07.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:39:12.465 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:39:15.852 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:39:16.724 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:39:19.274 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:39:30.223 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:39:30.223 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:39:30.262 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:39:31.953 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:41:32.654 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:41:37.812 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:41:38.879 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:41:40.563 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:41:42.650 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:41:42.650 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:41:42.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:41:42.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:41:43.448 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:41:49.113 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:41:50.466 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:41:51.941 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:41:51.941 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:41:51.941 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:41:51.984 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:41:54.095 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:43:56.513 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:43:59.155 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:43:59.896 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:44:01.104 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:44:02.307 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:44:02.307 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:44:02.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:44:02.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:44:04.003 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:44:06.339 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:44:07.286 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:44:08.388 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:44:08.388 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:44:08.388 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:44:08.426 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:44:10.429 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:46:17.857 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:46:18.736 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:46:24.839 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:46:26.088 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:46:29.829 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:46:29.829 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:46:29.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:46:30.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:46:32.406 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:46:34.071 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:46:34.956 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:46:35.984 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:46:35.984 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:46:35.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:46:36.024 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:46:36.802 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:48:37.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:48:39.313 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:48:45.549 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:48:47.488 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:48:49.266 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:48:49.266 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:48:49.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:48:49.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:48:50.907 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:48:53.465 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:48:56.141 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:48:58.001 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:48:58.002 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:48:58.002 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:48:58.051 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:48:59.342 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:51:00.821 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:51:04.896 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:51:08.494 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:51:09.862 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:51:11.260 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:51:12.462 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:51:12.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:51:12.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:51:15.101 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:51:16.904 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:51:18.225 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:51:20.750 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:51:20.750 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:51:20.750 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:51:20.787 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:51:22.018 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:53:23.110 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:53:24.836 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:53:26.971 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:53:31.500 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:53:34.505 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:53:34.505 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:53:34.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:53:34.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:53:36.736 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:53:38.196 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:53:40.996 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:53:42.911 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:53:44.133 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:53:44.133 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:53:44.173 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:53:44.755 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:55:47.497 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:55:48.419 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:55:50.699 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:55:52.754 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:55:55.505 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:55:55.505 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:55:55.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:55:55.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:55:57.431 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:55:59.172 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:56:01.013 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:56:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:56:02.810 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:56:02.811 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:56:02.849 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:56:04.694 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:58:07.380 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T08:58:09.091 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:58:10.384 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T08:58:11.531 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T08:58:14.374 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T08:58:14.374 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:58:14.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T08:58:14.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T08:58:16.131 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:58:17.388 INFO:teuthology.misc:Compressing logs... 2023-03-21T08:58:19.053 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T08:58:21.441 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T08:58:21.444 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T08:58:21.444 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T08:58:21.482 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T08:58:23.130 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:00:25.416 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:00:28.214 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:00:30.624 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:00:31.856 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:00:32.894 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:00:32.894 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:00:32.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:00:33.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:00:34.623 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:00:36.900 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:00:37.875 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:00:40.311 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:00:42.585 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:00:42.586 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:00:45.870 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:00:51.550 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:02:53.524 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:02:54.814 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:02:56.737 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:02:57.751 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:02:58.919 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:02:58.920 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:02:58.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:02:59.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:03:01.213 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:03:02.168 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:03:02.696 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:03:04.136 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:03:04.137 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:03:04.137 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:03:04.181 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:03:04.791 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:05:06.337 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:05:07.449 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:05:09.944 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:05:12.542 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:05:14.697 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:05:14.697 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:05:14.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:05:14.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:05:16.276 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:05:18.011 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:05:18.441 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:05:19.923 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:05:19.924 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:05:19.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:05:22.382 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:05:25.252 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:07:28.312 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:07:30.174 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:07:32.308 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:07:34.401 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:07:36.184 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:07:36.184 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:07:36.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:07:36.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:07:39.781 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:07:42.205 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:07:46.495 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:07:48.326 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:07:48.327 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:07:48.327 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:07:50.619 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:07:53.695 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:09:57.960 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:10:02.504 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:10:04.355 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:10:05.934 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:10:07.570 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:10:07.571 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:10:09.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:10:09.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:10:11.128 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:10:11.901 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:10:13.034 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:10:14.304 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:10:14.304 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:10:14.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:10:14.342 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:10:15.583 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:12:17.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:12:20.166 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:12:21.799 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:12:23.994 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:12:26.310 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:12:26.310 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:12:26.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:12:26.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:12:29.226 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:12:30.782 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:12:31.610 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:12:33.567 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:12:33.567 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:12:33.568 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:12:33.605 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:12:38.948 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:14:42.165 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:14:43.990 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:14:46.285 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:14:47.582 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:14:48.375 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:14:48.375 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:14:48.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:14:48.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:14:49.371 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:14:50.163 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:14:51.255 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:14:52.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:14:52.893 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:14:52.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:14:52.930 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:14:54.739 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:16:57.748 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:16:59.463 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:17:01.680 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:17:03.600 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:17:04.695 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:17:04.695 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:17:04.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:17:04.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:17:07.281 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:17:08.878 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:17:10.254 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:17:12.148 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:17:13.973 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:17:13.974 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:17:14.014 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:17:15.918 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:19:17.478 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:19:18.295 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:19:19.568 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:19:21.165 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:19:22.565 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:19:22.565 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:19:22.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:19:22.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:19:24.232 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:19:25.520 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:19:26.714 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:19:29.180 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:19:29.182 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:19:29.182 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:19:29.219 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:19:31.233 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:21:34.629 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:21:36.870 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:21:39.730 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:21:44.388 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:21:47.680 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:21:47.681 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:21:47.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:21:47.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:21:50.199 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:21:52.354 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:21:56.238 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:21:57.751 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:21:57.751 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:21:57.751 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:21:59.314 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:22:01.122 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:24:02.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:24:04.602 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:24:06.056 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:24:07.094 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:24:08.710 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:24:08.710 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:24:08.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:24:08.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:24:11.007 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:24:12.776 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:24:14.459 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:24:15.678 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:24:15.678 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:24:15.678 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:24:15.716 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:24:17.227 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:26:19.037 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:26:20.055 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:26:21.021 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:26:22.028 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:26:22.578 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:26:22.579 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:26:22.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:26:22.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:26:24.036 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:26:25.026 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:26:26.643 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:26:28.199 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:26:28.200 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:26:28.201 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:26:28.237 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:26:31.057 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:28:34.041 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:28:39.819 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:28:41.523 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:28:45.599 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:28:49.967 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:28:49.967 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:28:49.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:28:50.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:28:56.495 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:29:01.524 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:29:04.304 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:29:06.578 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:29:06.579 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:29:06.579 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:29:07.947 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:29:09.001 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:31:10.231 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:31:11.096 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:31:14.779 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:31:15.588 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:31:16.923 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:31:16.923 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:31:16.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:31:17.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:31:18.012 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:31:18.945 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:31:20.462 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:31:23.995 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:31:23.996 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:31:23.996 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:31:24.033 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:31:27.446 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:33:31.863 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:33:32.824 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:33:33.362 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:33:33.780 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:33:34.169 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:33:34.169 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:33:34.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:33:34.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:33:34.915 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:33:35.375 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:33:35.907 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:33:37.025 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:33:38.566 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:33:38.566 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:33:40.078 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:33:42.031 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:35:43.693 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:35:44.235 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:35:47.172 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:35:48.213 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:35:49.718 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:35:49.718 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:35:49.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:35:49.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:35:51.014 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:35:54.249 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:35:55.070 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:35:56.895 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:35:56.896 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:35:56.896 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:35:56.933 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:35:57.933 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:38:00.286 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:38:03.041 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:38:04.717 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:38:07.382 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:38:09.230 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:38:10.039 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:38:10.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:38:10.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:38:10.936 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:38:11.643 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:38:12.750 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:38:14.339 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:38:14.339 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:38:14.339 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:38:14.378 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:38:15.373 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:40:17.294 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:40:19.993 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:40:22.448 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:40:24.381 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:40:25.752 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:40:25.753 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:40:25.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:40:25.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:40:27.106 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:40:27.792 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:40:28.326 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:40:28.807 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:40:28.807 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:40:28.808 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:40:28.849 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:40:29.416 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:42:30.766 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:42:32.174 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:42:34.448 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:42:36.711 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:42:39.626 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:42:39.626 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:42:39.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:42:39.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:42:41.750 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:42:42.138 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:42:44.217 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:42:47.829 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:42:47.830 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:42:47.830 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:42:47.867 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:42:50.424 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:44:53.032 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:44:54.338 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:44:56.093 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:44:57.816 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:44:58.823 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:44:58.823 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:44:58.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:44:59.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:45:00.077 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:45:01.370 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:45:02.765 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:45:05.800 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:45:05.800 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:45:05.800 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:45:05.845 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:45:07.478 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:47:17.517 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:47:18.616 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:47:20.047 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:47:21.393 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:47:22.714 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:47:22.714 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:47:22.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:47:22.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:47:24.719 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:47:26.235 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:47:27.343 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:47:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:47:28.952 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:47:30.305 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:47:30.347 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:47:31.987 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:49:33.393 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:49:35.963 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:49:38.000 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:49:39.681 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:49:41.693 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:49:41.693 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:49:42.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:49:45.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:49:47.599 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:49:49.737 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:49:51.107 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:49:52.133 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:49:53.176 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:49:54.876 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:49:54.916 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:49:55.892 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:51:56.957 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:51:58.541 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:52:00.226 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:52:02.464 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:52:04.578 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:52:04.578 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:52:04.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:52:04.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:52:06.237 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:52:07.791 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:52:09.164 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:52:10.768 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:52:10.768 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:52:10.768 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:52:11.396 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:52:13.115 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:54:14.343 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:54:15.014 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:54:15.956 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:54:17.205 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:54:19.154 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:54:19.154 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:54:19.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:54:19.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:54:21.234 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:54:23.099 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:54:24.166 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:54:26.122 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:54:28.580 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:54:28.581 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:54:28.630 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:54:30.019 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:56:32.797 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:56:35.165 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:56:36.630 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:56:38.854 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:56:39.747 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:56:39.747 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:56:39.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:56:39.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:56:41.573 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:56:42.963 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:56:44.550 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:56:47.842 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:56:47.843 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:56:48.527 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:56:49.109 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:56:50.718 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:58:51.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T09:58:53.967 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:58:57.683 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T09:59:00.633 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T09:59:02.550 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T09:59:02.550 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:59:02.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T09:59:02.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T09:59:04.936 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:59:06.671 INFO:teuthology.misc:Compressing logs... 2023-03-21T09:59:07.443 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T09:59:08.969 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T09:59:08.969 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T09:59:08.969 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T09:59:09.018 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T09:59:12.436 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T10:01:14.508 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T10:01:17.796 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T10:01:19.145 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T10:01:20.354 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T10:01:25.726 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T10:01:25.726 INFO:teuthology.misc:Compressing logs... 2023-03-21T10:01:25.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T10:01:25.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T10:01:27.745 INFO:teuthology.misc:Compressing logs... 2023-03-21T10:01:29.764 INFO:teuthology.misc:Compressing logs... 2023-03-21T10:01:32.288 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T10:01:33.773 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-21T10:01:33.774 INFO:teuthology.orchestra.run.smithi129.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-21T10:01:33.774 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T10:01:35.225 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T10:01:37.997 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T10:03:42.298 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-21T10:03:44.858 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T10:03:47.796 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T10:03:49.326 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-21T10:03:51.186 INFO:teuthology.task.internal:roles: ubuntu@smithi129.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-21T10:03:51.186 INFO:teuthology.misc:Compressing logs... 2023-03-21T10:03:51.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi052.front.sepia.ceph.com' 2023-03-21T10:03:51.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-03-21T10:03:53.656 INFO:teuthology.misc:Compressing logs... 2023-03-21T10:03:55.785 INFO:teuthology.misc:Compressing logs... 2023-03-21T10:19:26.818 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-21T10:19:26.819 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi052 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-21T10:19:26.948 INFO:teuthology.kill:Killing Pids: {22971} 2023-03-21T10:19:26.974 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'} 2023-03-21T10:21:27.225 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-03-21T10:21:27.309 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-03-21T10:21:27.309 INFO:teuthology.nuke:Checking targets against current locks 2023-03-21T10:21:27.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-21T10:21:27.388 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-21T10:21:27.444 INFO:teuthology.orchestra.console:Power off smithi052 2023-03-21T10:21:27.461 INFO:teuthology.orchestra.console:Power off smithi129 2023-03-21T10:21:35.829 INFO:teuthology.orchestra.console:Power off for smithi052 completed 2023-03-21T10:21:35.982 INFO:teuthology.lock.ops:unlocked: smithi052.front.sepia.ceph.com 2023-03-21T10:21:39.953 INFO:teuthology.orchestra.console:Power off for smithi129 completed 2023-03-21T10:21:40.106 INFO:teuthology.lock.ops:unlocked: smithi129.front.sepia.ceph.com