2023-03-17T13:44:40.307 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-17T13:44:41.815 INFO:teuthology.lock.ops:Start node 'smithi177.front.sepia.ceph.com' reimaging 2023-03-17T13:44:43.317 INFO:teuthology.lock.ops:Updating [smithi177.front.sepia.ceph.com]: reset os type and version on server 2023-03-17T13:44:43.318 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2023-03-17T13:44:43.348 INFO:teuthology.lock.ops:Node 'smithi177.front.sepia.ceph.com' reimaging is complete 2023-03-17T13:44:44.855 INFO:teuthology.lock.ops:Start node 'smithi193.front.sepia.ceph.com' reimaging 2023-03-17T13:44:45.397 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: reset os type and version on server 2023-03-17T13:44:45.397 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2023-03-17T13:44:45.422 INFO:teuthology.lock.ops:Node 'smithi193.front.sepia.ceph.com' reimaging is complete 2023-03-17T13:44:46.737 INFO:teuthology.provision.fog.smithi177:Scheduling deploy of centos 8.stream 2023-03-17T13:44:53.990 INFO:teuthology.provision.fog.smithi193:Scheduling deploy of centos 8.stream 2023-03-17T13:44:55.557 INFO:teuthology.orchestra.console:Power off smithi177 2023-03-17T13:44:56.921 INFO:teuthology.orchestra.console:Power off smithi193 2023-03-17T13:45:06.835 INFO:teuthology.orchestra.console:Power off for smithi177 completed 2023-03-17T13:45:07.652 INFO:teuthology.orchestra.console:Power on smithi177 2023-03-17T13:45:10.952 INFO:teuthology.orchestra.console:Power off for smithi193 completed 2023-03-17T13:45:13.270 INFO:teuthology.orchestra.console:Power on smithi193 2023-03-17T13:45:15.034 INFO:teuthology.orchestra.console:Power on for smithi177 completed 2023-03-17T13:45:16.845 INFO:teuthology.provision.fog.smithi177:Waiting for deploy to finish 2023-03-17T13:45:19.068 INFO:teuthology.orchestra.console:Power on for smithi193 completed 2023-03-17T13:45:20.422 INFO:teuthology.provision.fog.smithi193:Waiting for deploy to finish 2023-03-17T13:48:06.405 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-03-17T13:48:10.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-03-17T13:48:16.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-03-17T13:48:20.457 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-03-17T13:48:25.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-03-17T13:48:30.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-03-17T13:48:36.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-03-17T13:48:41.349 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-03-17T13:48:56.965 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-03-17T13:49:02.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-03-17T13:49:23.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-03-17T13:49:32.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.177 2023-03-17T13:49:40.170 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-17T13:49:42.850 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-17T13:49:58.047 INFO:teuthology.provision.fog.smithi177:Node is ready 2023-03-17T13:49:59.719 INFO:teuthology.orchestra.run.smithi177.stdout:smithi177.front.sepia.ceph.com 2023-03-17T13:50:01.835 INFO:teuthology.provision.fog.smithi193:Node is ready 2023-03-17T13:50:01.852 INFO:teuthology.orchestra.run.smithi177.stdout:172.21.15.177 smithi177.front.sepia.ceph.com smithi177 2023-03-17T13:50:03.416 INFO:teuthology.orchestra.run.smithi193.stdout:smithi193.front.sepia.ceph.com 2023-03-17T13:50:04.282 INFO:teuthology.orchestra.run.smithi193.stdout:172.21.15.193 smithi193.front.sepia.ceph.com smithi193 2023-03-17T13:50:06.378 INFO:teuthology.provision.fog.smithi177:Deploy complete! 2023-03-17T13:50:07.305 INFO:teuthology.provision.fog.smithi193:Deploy complete! 2023-03-17T13:50:09.393 INFO:teuthology.lock.ops:Checking smithi177.front.sepia.ceph.com 2023-03-17T13:50:10.628 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-17T13:50:10.650 INFO:teuthology.lock.ops:Checking smithi193.front.sepia.ceph.com 2023-03-17T13:50:12.085 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-17T13:50:12.118 INFO:teuthology.lock.ops:Updating [smithi177.front.sepia.ceph.com]: set os type and version on server 2023-03-17T13:50:13.773 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-17T13:50:18.000 INFO:teuthology.orchestra.run.smithi177.stdout:x86_64 2023-03-17T13:50:20.425 INFO:teuthology.orchestra.run.smithi177.stdout:NAME="CentOS Stream" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION="8" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:ID="centos" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:ID_LIKE="rhel fedora" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:VERSION_ID="8" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:PLATFORM_ID="platform:el8" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:ANSI_COLOR="0;31" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:HOME_URL="https://centos.org/" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-17T13:50:23.714 INFO:teuthology.orchestra.run.smithi177.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-17T13:50:23.715 INFO:teuthology.lock.ops:Updating smithi177.front.sepia.ceph.com on lock server 2023-03-17T13:50:23.739 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: set os type and version on server 2023-03-17T13:50:27.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-17T13:50:27.388 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2023-03-17T13:50:31.798 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-17T13:50:34.361 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-17T13:50:34.362 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2023-03-17T13:50:34.474 INFO:teuthology.dispatcher.supervisor:Running job 7209208 2023-03-17T13:50:35.537 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209208 --name yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi --description rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/dashboard} -- /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209208/orig.config.yaml 2023-03-17T13:50:35.543 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209208 2023-03-17T13:50:37.582 INFO:teuthology.dispatcher.supervisor:Job PID: 18386 2023-03-17T13:50:37.583 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-03-18T01:51:29.405 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T01:51:32.109 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T01:51:34.249 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) KeyError: 'status' 2023-03-18T01:51:35.519 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T01:51:36.679 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T01:51:36.679 INFO:teuthology.misc:Compressing logs... 2023-03-18T01:51:37.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T01:51:37.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T01:51:39.351 INFO:teuthology.misc:Compressing logs... 2023-03-18T01:51:40.602 INFO:teuthology.misc:Compressing logs... 2023-03-18T01:52:05.029 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-03-18T01:52:10.781 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/rgw.ceph.client.0.log: file size changed while zipping 2023-03-18T01:52:27.116 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-03-18T01:52:51.034 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-03-18T01:53:00.288 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-03-18T01:53:03.593 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-03-18T01:53:14.517 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-03-18T01:53:16.514 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-03-18T01:54:07.429 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-03-18T01:54:16.524 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-03-18T01:54:19.063 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.z.log: file size changed while zipping 2023-03-18T01:55:05.649 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T01:55:06.583 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T01:57:07.921 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T01:57:09.257 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T01:57:11.323 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T01:57:13.779 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T01:57:14.891 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T01:57:14.891 INFO:teuthology.misc:Compressing logs... 2023-03-18T01:57:14.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T01:57:15.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T01:57:19.749 INFO:teuthology.misc:Compressing logs... 2023-03-18T01:57:21.003 INFO:teuthology.misc:Compressing logs... 2023-03-18T01:58:07.944 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T01:58:09.526 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:00:10.842 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:00:12.518 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:00:14.187 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:00:19.131 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:00:20.883 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:00:20.884 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:00:20.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:00:21.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:00:22.863 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:00:27.337 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:01:15.528 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:01:19.448 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:03:21.424 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:03:22.346 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:03:23.155 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:03:23.837 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:03:24.947 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:03:26.463 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:03:26.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:03:26.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:03:27.424 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:03:27.854 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:04:15.116 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:04:17.835 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:06:21.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:06:23.416 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:06:24.812 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:06:27.081 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:06:28.695 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:06:28.695 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:06:28.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:06:28.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:06:30.234 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:06:32.509 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:07:20.178 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:07:20.888 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:09:22.197 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:09:24.000 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:09:25.220 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:09:27.124 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:09:28.922 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:09:28.922 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:09:28.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:09:29.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:09:31.602 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:09:33.149 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:10:21.042 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:10:22.353 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:12:25.009 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:12:27.608 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:12:28.948 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:12:30.399 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:12:30.993 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:12:30.994 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:12:30.994 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:12:32.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:12:33.099 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:12:33.945 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:13:22.044 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:13:23.333 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:15:24.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:15:25.549 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:15:27.233 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:15:28.983 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:15:30.065 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:15:30.065 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:15:30.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:15:30.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:15:32.380 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:15:33.975 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:15:35.807 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-18T02:15:37.358 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-18T02:15:37.358 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-18T02:15:37.358 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-18T02:15:37.403 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:15:39.427 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:17:41.334 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:17:41.962 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:17:44.283 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:17:45.998 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:17:47.619 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:17:47.619 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:17:47.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:17:47.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:17:49.881 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:18:00.004 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:18:01.094 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-18T02:18:03.392 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-18T02:18:03.392 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-18T02:18:03.393 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-18T02:18:03.463 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:18:06.533 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:20:08.407 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:20:10.198 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:20:12.187 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:20:14.784 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:20:15.982 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:20:15.982 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:20:15.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:20:16.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:20:17.277 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:20:18.339 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:20:19.552 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-18T02:20:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-18T02:20:20.870 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-18T02:20:20.871 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-18T02:20:20.916 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:20:22.755 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:22:24.285 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:22:26.209 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:22:31.069 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:22:32.821 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:22:34.299 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:22:34.299 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:22:34.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:22:34.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:22:35.595 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:22:36.348 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:22:37.577 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-18T02:22:39.099 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-18T02:22:39.100 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-18T02:22:39.100 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-18T02:22:39.152 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:22:40.710 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:24:41.887 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:24:46.638 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:24:48.568 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:24:49.805 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:24:51.017 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:24:51.017 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:24:51.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:24:51.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:24:52.444 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:24:53.007 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:24:53.613 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-18T02:24:54.084 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-18T02:24:54.491 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-18T02:24:54.492 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-18T02:24:54.538 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:24:55.248 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:26:56.209 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-18T02:26:57.918 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:27:04.321 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:27:04.883 INFO:teuthology.task.internal:roles: ubuntu@smithi177.front.sepia.ceph.com - ['mgr.x', 'mon.a', 'mon.c', 'mds.a', 'mds.c', 'osd.0', 'client.0'] 2023-03-18T02:27:06.459 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mgr.y', 'mgr.z', 'mon.b', 'mds.b', 'osd.1', 'osd.2', 'osd.3', 'client.1'] 2023-03-18T02:27:06.459 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:27:06.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi177.front.sepia.ceph.com' 2023-03-18T02:27:06.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-03-18T02:27:07.707 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:27:08.854 INFO:teuthology.misc:Compressing logs... 2023-03-18T02:27:10.045 INFO:teuthology.orchestra.run.smithi177.stderr:gzip: /var/log/ceph/ceph-mgr.x.log.gz already exists; not overwritten 2023-03-18T02:27:15.945 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.y.log.gz already exists; not overwritten 2023-03-18T02:27:15.945 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/ceph-mgr.z.log.gz already exists; not overwritten 2023-03-18T02:27:15.945 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi177 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-18T02:27:15.989 INFO:teuthology.kill:Killing Pids: {18386} 2023-03-18T02:27:28.824 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results', 'teuthology_job_results_created'} 2023-03-18T02:29:32.698 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-03-18T02:29:33.307 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-03-18T02:29:34.362 INFO:teuthology.nuke:Checking targets against current locks 2023-03-18T02:29:34.417 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-18T02:29:35.143 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-18T02:29:37.383 INFO:teuthology.orchestra.console:Power off smithi177 2023-03-18T02:29:39.300 INFO:teuthology.orchestra.console:Power off smithi193 2023-03-18T02:29:49.325 INFO:teuthology.orchestra.console:Power off for smithi177 completed 2023-03-18T02:29:50.641 INFO:teuthology.orchestra.console:Power off for smithi193 completed 2023-03-18T02:29:50.801 INFO:teuthology.lock.ops:unlocked: smithi193.front.sepia.ceph.com 2023-03-18T02:29:52.898 INFO:teuthology.lock.ops:unlocked: smithi177.front.sepia.ceph.com