2023-03-15T21:01:23.639 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-15T21:01:25.453 INFO:teuthology.lock.ops:Start node 'smithi146.front.sepia.ceph.com' reimaging 2023-03-15T21:01:29.034 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: reset os type and version on server 2023-03-15T21:01:33.878 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2023-03-15T21:01:35.346 INFO:teuthology.lock.ops:Node 'smithi146.front.sepia.ceph.com' reimaging is complete 2023-03-15T21:01:36.611 INFO:teuthology.lock.ops:Start node 'smithi188.front.sepia.ceph.com' reimaging 2023-03-15T21:01:36.611 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: reset os type and version on server 2023-03-15T21:01:36.611 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2023-03-15T21:01:36.645 INFO:teuthology.lock.ops:Node 'smithi188.front.sepia.ceph.com' reimaging is complete 2023-03-15T21:01:37.836 INFO:teuthology.provision.fog.smithi146:Scheduling deploy of centos 8.stream 2023-03-15T21:01:42.692 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of centos 8.stream 2023-03-15T21:01:47.590 INFO:teuthology.orchestra.console:Power off smithi146 2023-03-15T21:01:49.623 INFO:teuthology.orchestra.console:Power off smithi188 2023-03-15T21:01:59.481 INFO:teuthology.orchestra.console:Power off for smithi146 completed 2023-03-15T21:02:00.134 INFO:teuthology.orchestra.console:Power on smithi146 2023-03-15T21:02:03.598 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2023-03-15T21:02:04.454 INFO:teuthology.orchestra.console:Power on smithi188 2023-03-15T21:02:06.800 INFO:teuthology.orchestra.console:Power on for smithi146 completed 2023-03-15T21:02:08.380 INFO:teuthology.provision.fog.smithi146:Waiting for deploy to finish 2023-03-15T21:02:14.947 INFO:teuthology.orchestra.console:Power on for smithi188 completed 2023-03-15T21:02:15.793 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish 2023-03-15T21:05:08.613 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188 2023-03-15T21:05:14.885 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-03-15T21:05:20.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188 2023-03-15T21:05:25.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-03-15T21:05:30.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188 2023-03-15T21:05:34.469 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-03-15T21:05:40.485 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188 2023-03-15T21:05:45.733 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-03-15T21:05:59.429 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.146 2023-03-15T21:06:50.376 WARNING:teuthology.provision.fog:timed out 2023-03-15T21:06:58.084 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-15T21:07:01.848 INFO:teuthology.provision.fog.smithi188:Node is ready 2023-03-15T21:07:04.067 INFO:teuthology.orchestra.run.smithi188.stdout:smithi188.front.sepia.ceph.com 2023-03-15T21:07:06.076 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.15.188 smithi188.front.sepia.ceph.com smithi188 2023-03-15T21:07:08.663 INFO:teuthology.provision.fog.smithi188:Deploy complete! 2023-03-15T21:07:12.659 WARNING:teuthology.provision.fog:timed out 2023-03-15T21:07:19.005 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-15T21:07:26.413 INFO:teuthology.provision.fog.smithi146:Node is ready 2023-03-15T21:07:27.994 INFO:teuthology.orchestra.run.smithi146.stdout:smithi146.front.sepia.ceph.com 2023-03-15T21:07:29.189 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.15.146 smithi146.front.sepia.ceph.com smithi146 2023-03-15T21:07:33.403 INFO:teuthology.provision.fog.smithi146:Deploy complete! 2023-03-15T21:07:35.018 INFO:teuthology.lock.ops:Checking smithi146.front.sepia.ceph.com 2023-03-15T21:07:36.540 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-15T21:07:36.562 INFO:teuthology.lock.ops:Checking smithi188.front.sepia.ceph.com 2023-03-15T21:07:41.066 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-15T21:07:41.088 INFO:teuthology.lock.ops:Updating [smithi146.front.sepia.ceph.com]: set os type and version on server 2023-03-15T21:07:43.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-15T21:07:43.208 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2023-03-15T21:07:47.368 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2023-03-15T21:07:50.686 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="8" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="8" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el8" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-15T21:07:50.687 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-15T21:07:50.688 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2023-03-15T21:07:52.677 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: set os type and version on server 2023-03-15T21:07:53.019 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-15T21:07:53.266 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2023-03-15T21:07:53.872 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2023-03-15T21:07:54.448 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-15T21:07:54.449 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/" 2023-03-15T21:07:54.934 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-15T21:07:54.934 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-15T21:07:54.934 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-15T21:07:55.259 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2023-03-15T21:07:55.374 INFO:teuthology.dispatcher.supervisor:Running job 7208340 2023-03-15T21:08:00.912 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-14_21:32:35-fs-quincy-release-distro-default-smithi/7208340 --name yuriw-2023-03-14_21:32:35-fs-quincy-release-distro-default-smithi --description fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} -- /home/teuthworker/archive/yuriw-2023-03-14_21:32:35-fs-quincy-release-distro-default-smithi/7208340/orig.config.yaml 2023-03-15T21:08:00.918 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-14_21:32:35-fs-quincy-release-distro-default-smithi/7208340 2023-03-15T21:08:05.595 INFO:teuthology.dispatcher.supervisor:Job PID: 914 2023-03-15T21:08:05.596 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-03-16T09:08:56.955 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:08:59.477 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:09:01.365 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) KeyError: 'status' 2023-03-16T09:09:02.620 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:09:04.792 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:09:04.792 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:09:04.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:09:05.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:09:06.376 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:09:07.843 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:09:23.289 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-osd.5.log: file size changed while zipping 2023-03-16T09:09:26.348 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log: file size changed while zipping 2023-03-16T09:09:34.857 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mon.a.log: file size changed while zipping 2023-03-16T09:09:39.057 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-osd.7.log: file size changed while zipping 2023-03-16T09:09:46.644 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-osd.2.log: file size changed while zipping 2023-03-16T09:09:52.056 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2023-03-16T09:09:54.219 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log: file size changed while zipping 2023-03-16T09:09:59.194 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-osd.1.log: file size changed while zipping 2023-03-16T09:10:01.007 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2023-03-16T09:10:01.648 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mgr.x.log: file size changed while zipping 2023-03-16T09:10:05.220 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mgr.y.log: file size changed while zipping 2023-03-16T09:10:10.217 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log: file size changed while zipping 2023-03-16T09:10:13.296 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-osd.6.log: file size changed while zipping 2023-03-16T09:10:22.313 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-osd.3.log: file size changed while zipping 2023-03-16T09:10:27.147 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-osd.4.log: file size changed while zipping 2023-03-16T09:10:34.838 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-osd.0.log: file size changed while zipping 2023-03-16T09:10:36.357 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log: file size changed while zipping 2023-03-16T09:10:57.465 INFO:teuthology.orchestra.run.smithi146.stderr:tar: .: file changed as we read it 2023-03-16T09:11:14.528 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:11:16.807 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:13:18.885 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:13:23.360 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:13:24.757 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:13:26.230 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:13:27.271 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:13:27.271 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:13:27.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:13:27.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:13:32.260 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:13:34.438 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:13:35.159 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:13:36.029 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:13:37.069 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:13:37.069 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:13:37.069 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:13:37.697 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:13:38.141 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:15:39.173 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:15:40.467 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:15:41.631 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:15:42.517 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:15:44.066 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:15:44.066 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:15:44.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:15:44.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:15:47.685 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:15:49.472 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:15:50.336 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:15:52.037 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:15:52.038 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:15:52.038 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:15:52.039 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:15:52.092 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:15:53.811 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:17:55.444 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:17:56.560 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:17:57.773 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:18:02.675 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:18:04.005 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:18:04.006 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:18:04.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:18:04.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:18:06.490 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:18:08.257 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:18:09.210 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:18:09.972 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:18:09.973 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:18:09.974 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:18:09.974 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:18:10.031 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:18:10.847 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:20:13.053 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:20:15.407 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:20:17.938 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:20:19.568 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:20:21.738 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:20:22.339 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:20:22.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:20:22.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:20:24.138 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:20:26.589 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:20:28.210 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:20:30.189 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:20:31.325 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:20:31.325 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:20:31.325 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:20:31.379 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:20:33.087 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:22:35.341 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:22:36.115 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:22:37.509 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:22:40.174 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:22:42.289 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:22:42.289 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:22:42.289 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:22:43.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:22:46.458 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:22:47.657 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:22:48.598 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:22:51.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:22:51.288 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:22:51.288 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:22:51.289 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:22:51.896 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:22:53.243 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:24:54.873 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:24:56.178 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:24:56.951 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:24:58.188 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:24:59.033 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:24:59.033 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:24:59.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:24:59.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:25:00.918 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:25:01.977 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:25:03.424 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:25:04.107 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:25:04.108 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:25:04.109 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:25:04.109 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:25:04.161 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:25:06.046 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:27:12.034 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:27:17.129 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:27:17.876 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:27:19.124 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:27:20.538 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:27:20.538 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:27:20.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:27:21.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:27:22.414 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:27:24.126 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:27:28.759 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:27:31.961 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:27:31.962 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:27:31.962 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:27:31.962 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:27:32.016 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:27:35.257 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:29:36.644 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-16T09:29:38.445 INFO:teuthology.kill:Killing Pids: {914} 2023-03-16T09:29:39.608 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:29:40.813 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-16T09:29:42.514 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-16T09:29:42.514 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:29:42.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2023-03-16T09:29:42.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com' 2023-03-16T09:29:46.050 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:29:49.989 INFO:teuthology.misc:Compressing logs... 2023-03-16T09:29:51.271 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.b.log.gz already exists; not overwritten 2023-03-16T09:29:52.697 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/ceph-mds.d.log.gz already exists; not overwritten 2023-03-16T09:29:52.699 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.a.log.gz already exists; not overwritten 2023-03-16T09:29:52.699 INFO:teuthology.orchestra.run.smithi146.stderr:gzip: /var/log/ceph/ceph-mds.c.log.gz already exists; not overwritten 2023-03-16T09:29:52.699 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi146 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-16T09:29:52.752 INFO:teuthology.kill:No teuthology processes running 2023-03-16T09:29:54.167 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"]) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results', 'teuthology_job_results_total'} 2023-03-16T09:31:54.407 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-03-16T09:31:55.362 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-03-16T09:31:57.047 INFO:teuthology.nuke:Checking targets against current locks 2023-03-16T09:31:57.107 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-16T09:32:00.993 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-16T09:32:02.158 INFO:teuthology.orchestra.console:Power off smithi146 2023-03-16T09:32:03.747 INFO:teuthology.orchestra.console:Power off smithi188 2023-03-16T09:32:13.872 INFO:teuthology.orchestra.console:Power off for smithi146 completed 2023-03-16T09:32:16.728 INFO:teuthology.lock.ops:unlocked: smithi146.front.sepia.ceph.com 2023-03-16T09:32:18.019 INFO:teuthology.orchestra.console:Power off for smithi188 completed 2023-03-16T09:32:19.768 INFO:teuthology.lock.ops:unlocked: smithi188.front.sepia.ceph.com