2023-03-16T01:37:08.196 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f
2023-03-16T01:37:10.043 INFO:teuthology.lock.ops:Start node 'smithi090.front.sepia.ceph.com' reimaging
2023-03-16T01:37:12.018 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T01:37:12.018 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-03-16T01:37:12.043 INFO:teuthology.lock.ops:Node 'smithi090.front.sepia.ceph.com' reimaging is complete
2023-03-16T01:37:12.938 INFO:teuthology.lock.ops:Start node 'smithi101.front.sepia.ceph.com' reimaging
2023-03-16T01:37:13.634 INFO:teuthology.lock.ops:Updating [smithi101.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T01:37:13.635 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server
2023-03-16T01:37:13.718 INFO:teuthology.lock.ops:Node 'smithi101.front.sepia.ceph.com' reimaging is complete
2023-03-16T01:37:14.681 INFO:teuthology.lock.ops:Start node 'smithi193.front.sepia.ceph.com' reimaging
2023-03-16T01:37:15.012 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: reset os type and version on server
2023-03-16T01:37:15.012 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server
2023-03-16T01:37:16.100 INFO:teuthology.lock.ops:Node 'smithi193.front.sepia.ceph.com' reimaging is complete
2023-03-16T01:37:17.761 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of rhel 8.6
2023-03-16T01:37:18.638 INFO:teuthology.provision.fog.smithi101:Scheduling deploy of rhel 8.6
2023-03-16T01:37:19.422 INFO:teuthology.provision.fog.smithi193:Scheduling deploy of rhel 8.6
2023-03-16T01:37:21.051 INFO:teuthology.orchestra.console:Power off smithi090
2023-03-16T01:37:23.055 INFO:teuthology.orchestra.console:Power off smithi101
2023-03-16T01:37:25.581 INFO:teuthology.orchestra.console:Power off smithi193
2023-03-16T01:37:31.870 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2023-03-16T01:37:34.664 INFO:teuthology.orchestra.console:Power on smithi090
2023-03-16T01:37:36.523 INFO:teuthology.orchestra.console:Power off for smithi101 completed
2023-03-16T01:37:38.745 INFO:teuthology.orchestra.console:Power off for smithi193 completed
2023-03-16T01:37:41.033 INFO:teuthology.orchestra.console:Power on smithi101
2023-03-16T01:37:41.134 INFO:teuthology.orchestra.console:Power on smithi193
2023-03-16T01:37:42.688 INFO:teuthology.orchestra.console:Power on for smithi090 completed
2023-03-16T01:37:43.797 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2023-03-16T01:37:45.287 INFO:teuthology.orchestra.console:Power on for smithi101 completed
2023-03-16T01:37:46.603 INFO:teuthology.provision.fog.smithi101:Waiting for deploy to finish
2023-03-16T01:37:48.220 INFO:teuthology.orchestra.console:Power on for smithi193 completed
2023-03-16T01:37:49.627 INFO:teuthology.provision.fog.smithi193:Waiting for deploy to finish
2023-03-16T01:40:52.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.101
2023-03-16T01:40:55.429 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-16T01:41:02.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.101
2023-03-16T01:41:04.407 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-16T01:41:07.461 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-16T01:41:13.477 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.101
2023-03-16T01:41:15.012 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-16T01:41:18.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-16T01:41:29.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-16T01:41:44.581 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.101
2023-03-16T01:41:48.414 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-16T01:41:54.309 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-16T01:41:58.693 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.101
2023-03-16T01:41:59.285 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-16T01:42:04.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.193
2023-03-16T01:42:05.671 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T01:42:05.976 INFO:teuthology.provision.fog.smithi101:Node is ready
2023-03-16T01:42:09.558 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-03-16T01:42:09.577 INFO:teuthology.orchestra.run.smithi101.stdout:smithi101.front.sepia.ceph.com
2023-03-16T01:42:20.463 INFO:teuthology.orchestra.run.smithi101.stdout:172.21.15.101 smithi101.front.sepia.ceph.com smithi101
2023-03-16T01:42:23.083 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T01:42:24.945 INFO:teuthology.provision.fog.smithi101:Deploy complete!
2023-03-16T01:42:26.222 INFO:teuthology.provision.fog.smithi193:Node is ready
2023-03-16T01:42:26.241 INFO:teuthology.orchestra.run.smithi193.stdout:smithi193.front.sepia.ceph.com
2023-03-16T01:42:28.082 INFO:teuthology.orchestra.run.smithi193.stdout:172.21.15.193 smithi193.front.sepia.ceph.com smithi193
2023-03-16T01:42:30.595 INFO:teuthology.provision.fog.smithi193:Deploy complete!
2023-03-16T01:42:52.452 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-16T01:42:53.282 INFO:teuthology.provision.fog.smithi090:Node is ready
2023-03-16T01:42:55.931 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com
2023-03-16T01:42:57.613 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090
2023-03-16T01:43:01.208 INFO:teuthology.provision.fog.smithi090:Deploy complete!
2023-03-16T01:43:02.480 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com
2023-03-16T01:43:04.263 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T01:43:04.295 INFO:teuthology.lock.ops:Checking smithi101.front.sepia.ceph.com
2023-03-16T01:43:06.011 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T01:43:06.036 INFO:teuthology.lock.ops:Checking smithi193.front.sepia.ceph.com
2023-03-16T01:43:07.579 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-16T01:43:08.504 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server
2023-03-16T01:43:09.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T01:43:10.779 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64
2023-03-16T01:43:11.979 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Red Hat Enterprise Linux"
2023-03-16T01:43:13.424 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8.6 (Ootpa)"
2023-03-16T01:43:13.424 INFO:teuthology.orchestra.run.smithi090.stdout:ID="rhel"
2023-03-16T01:43:13.424 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="fedora"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8.6"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.redhat.com/"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-16T01:43:13.425 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-16T01:43:13.426 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-16T01:43:13.427 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-03-16T01:43:13.444 INFO:teuthology.lock.ops:Updating [smithi101.front.sepia.ceph.com]: set os type and version on server
2023-03-16T01:43:14.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T01:43:15.062 INFO:teuthology.orchestra.run.smithi101.stdout:x86_64
2023-03-16T01:43:15.811 INFO:teuthology.orchestra.run.smithi101.stdout:NAME="Red Hat Enterprise Linux"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION="8.6 (Ootpa)"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:ID="rhel"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:ID_LIKE="fedora"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:VERSION_ID="8.6"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:PLATFORM_ID="platform:el8"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:ANSI_COLOR="0;31"
2023-03-16T01:43:16.685 INFO:teuthology.orchestra.run.smithi101.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:HOME_URL="https://www.redhat.com/"
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-16T01:43:16.686 INFO:teuthology.orchestra.run.smithi101.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-16T01:43:16.687 INFO:teuthology.lock.ops:Updating smithi101.front.sepia.ceph.com on lock server
2023-03-16T01:43:16.711 INFO:teuthology.lock.ops:Updating [smithi193.front.sepia.ceph.com]: set os type and version on server
2023-03-16T01:43:17.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T01:43:18.167 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64
2023-03-16T01:43:19.109 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="Red Hat Enterprise Linux"
2023-03-16T01:43:20.142 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8.6 (Ootpa)"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:ID="rhel"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="fedora"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8.6"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://www.redhat.com/"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-16T01:43:20.143 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-16T01:43:20.616 INFO:teuthology.orchestra.run.smithi193.stdout:
2023-03-16T01:43:20.616 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-16T01:43:20.616 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-03-16T01:43:20.616 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-16T01:43:20.616 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-03-16T01:43:20.617 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server
2023-03-16T01:43:20.738 INFO:teuthology.dispatcher.supervisor:Running job 7209805
2023-03-16T01:43:21.781 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_xiubli@teuthology --archive /home/teuthworker/archive/xiubli-2023-03-16_00:26:19-fs:workload-wip-20230315-1512-unlink-distro-default-smithi/7209805 --name xiubli-2023-03-16_00:26:19-fs:workload-wip-20230315-1512-unlink-distro-default-smithi --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} -- /home/teuthworker/archive/xiubli-2023-03-16_00:26:19-fs:workload-wip-20230315-1512-unlink-distro-default-smithi/7209805/orig.config.yaml
2023-03-16T01:43:21.786 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/xiubli-2023-03-16_00:26:19-fs:workload-wip-20230315-1512-unlink-distro-default-smithi/7209805
2023-03-16T01:43:22.873 INFO:teuthology.dispatcher.supervisor:Job PID: 30501
2023-03-16T01:43:22.873 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-16T13:44:14.369 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T13:44:16.364 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:44:19.914 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
KeyError: 'status'
2023-03-16T13:44:20.460 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T13:44:22.556 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T13:44:22.556 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T13:44:22.556 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:44:22.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T13:44:22.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T13:44:24.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T13:44:25.903 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:44:29.749 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:45:00.943 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2023-03-16T13:45:20.116 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-16T13:45:21.736 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.e.log: file size changed while zipping
2023-03-16T13:45:22.068 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-03-16T13:45:35.684 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2023-03-16T13:45:53.712 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2023-03-16T13:45:57.979 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-03-16T13:46:18.334 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2023-03-16T13:46:20.594 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-16T13:46:24.705 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mgr.x.log: file size changed while zipping
2023-03-16T13:46:28.308 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2023-03-16T13:46:30.264 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log: file size changed while zipping
2023-03-16T13:46:55.274 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-16T13:46:58.560 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.f.log: file size changed while zipping
2023-03-16T13:46:58.820 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.c.log: file size changed while zipping
2023-03-16T13:47:00.278 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.i.log: file size changed while zipping
2023-03-16T13:47:02.941 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-16T13:47:05.767 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-osd.9.log: file size changed while zipping
2023-03-16T13:47:09.216 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2023-03-16T13:47:12.330 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2023-03-16T13:47:21.820 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.b.log: file size changed while zipping
2023-03-16T13:47:33.351 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2023-03-16T13:49:24.292 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:49:25.269 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:51:26.572 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T13:51:29.617 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:51:30.740 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:51:32.002 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T13:51:33.437 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T13:51:34.113 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T13:51:34.113 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:51:34.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T13:51:34.339 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T13:51:35.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T13:51:37.932 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:51:42.289 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:51:45.391 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:51:49.665 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:51:49.667 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:51:53.075 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:51:53.075 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:51:53.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T13:51:53.159 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:51:54.599 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:53:56.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T13:53:57.123 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:53:57.994 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:54:00.693 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T13:54:02.954 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T13:54:02.954 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T13:54:03.686 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:54:06.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T13:54:08.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T13:54:10.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T13:54:11.823 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:54:13.767 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:54:14.732 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:54:16.381 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:54:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:54:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:54:16.383 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:54:16.383 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:54:16.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T13:54:18.108 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:54:20.471 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:56:21.993 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T13:56:23.094 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:56:24.437 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:56:25.802 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T13:56:27.349 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T13:56:27.349 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T13:56:27.349 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:56:27.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T13:56:27.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T13:56:34.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T13:56:35.685 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:56:37.028 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:56:38.423 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:56:40.592 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T13:56:42.107 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:56:42.109 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:56:42.109 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:56:42.109 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:56:42.109 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:56:42.109 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T13:56:42.191 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:56:45.108 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:58:48.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T13:58:49.829 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:58:51.697 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T13:58:55.095 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T13:58:56.106 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T13:58:56.106 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T13:58:56.106 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:58:57.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T13:58:57.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T13:59:00.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T13:59:02.644 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:59:05.491 INFO:teuthology.misc:Compressing logs...
2023-03-16T13:59:08.962 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:59:12.057 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:59:12.059 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:59:14.735 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:59:15.150 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T13:59:18.206 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T13:59:18.206 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T13:59:18.206 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T13:59:18.277 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T13:59:20.129 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:01:21.600 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T14:01:23.611 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:01:25.561 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:01:27.011 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T14:01:29.585 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T14:01:29.586 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T14:01:29.586 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:01:29.586 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T14:01:29.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T14:01:31.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T14:01:33.566 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:01:34.706 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:01:35.944 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:01:37.750 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T14:01:37.750 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:01:37.752 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:01:37.753 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:01:38.631 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:01:38.631 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:01:38.632 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T14:01:38.704 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:01:39.986 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:03:41.489 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T14:03:41.856 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:03:42.503 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:03:43.194 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T14:03:44.029 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T14:03:44.029 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T14:03:44.029 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:03:44.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T14:03:44.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T14:03:45.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T14:03:47.005 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:03:49.281 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:03:49.416 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:03:50.598 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:03:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:03:50.600 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:03:50.600 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T14:03:50.601 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:03:50.601 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T14:03:50.601 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:03:50.674 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:03:51.375 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:05:52.342 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T14:05:53.971 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:05:55.894 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:05:57.506 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T14:05:59.031 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T14:05:59.031 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T14:06:00.432 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:06:00.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T14:06:00.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T14:06:02.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T14:06:04.445 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:06:05.701 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:06:07.202 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:06:08.532 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T14:06:08.532 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:06:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:06:08.534 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:06:08.535 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:06:08.535 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:06:08.535 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T14:06:10.242 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:06:11.441 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:08:14.564 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T14:08:15.832 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:08:17.326 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:08:18.092 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T14:08:18.856 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T14:08:20.446 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T14:08:20.446 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:08:20.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T14:08:20.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T14:08:22.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T14:08:24.305 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:08:25.358 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:08:26.333 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:08:28.862 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T14:08:28.863 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:08:28.864 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:08:28.865 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:08:28.865 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:08:28.865 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:08:28.865 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T14:08:28.936 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:08:30.669 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:10:32.273 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T14:10:33.693 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:10:34.026 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:10:34.909 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T14:10:36.536 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T14:10:36.536 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T14:10:36.536 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:10:36.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T14:10:36.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T14:10:37.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T14:10:39.018 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:10:39.718 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:10:40.294 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:10:41.097 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T14:10:41.097 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:10:41.101 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:10:41.939 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:10:43.184 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:10:43.184 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:10:43.810 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T14:10:43.886 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:10:45.020 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:12:46.815 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-16T14:12:47.124 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:12:48.450 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:12:49.553 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
2023-03-16T14:12:50.199 INFO:teuthology.task.internal:roles: ubuntu@smithi101.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
2023-03-16T14:12:50.199 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
2023-03-16T14:12:50.200 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:12:50.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-03-16T14:12:50.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi101.front.sepia.ceph.com'
2023-03-16T14:12:51.692 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com'
2023-03-16T14:12:53.408 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:12:53.888 INFO:teuthology.misc:Compressing logs...
2023-03-16T14:12:55.158 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:12:55.658 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-mds.k.log.gz already exists;	not overwritten
2023-03-16T14:12:55.660 INFO:teuthology.orchestra.run.smithi101.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:12:55.661 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:12:55.662 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:12:55.662 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-16T14:12:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/2ebc43e2-c3a0-11ed-9afb-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-16T14:12:56.698 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi090 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-16T14:12:59.432 INFO:teuthology.kill:Killing Pids: {30501}
2023-03-16T14:13:00.968 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_total', 'teuthology_job_results_created'}
2023-03-16T14:15:01.203 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-16T14:15:01.692 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-16T14:15:01.827 INFO:teuthology.nuke:Checking targets against current locks
2023-03-16T14:15:01.897 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-16T14:15:02.875 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-16T14:15:03.219 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-16T14:15:04.224 INFO:teuthology.orchestra.console:Power off smithi090
2023-03-16T14:15:05.270 INFO:teuthology.orchestra.console:Power off smithi193
2023-03-16T14:15:06.959 INFO:teuthology.orchestra.console:Power off smithi101
2023-03-16T14:15:16.367 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2023-03-16T14:15:16.720 INFO:teuthology.orchestra.console:Power off for smithi193 completed
2023-03-16T14:15:16.720 INFO:teuthology.orchestra.console:Power off for smithi101 completed
2023-03-16T14:15:16.899 INFO:teuthology.lock.ops:unlocked: smithi101.front.sepia.ceph.com
2023-03-16T14:15:17.996 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com
2023-03-16T14:15:19.448 INFO:teuthology.lock.ops:unlocked: smithi193.front.sepia.ceph.com