2023-03-18T03:22:36.865 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f
2023-03-18T03:22:38.043 INFO:teuthology.lock.ops:Start node 'smithi086.front.sepia.ceph.com' reimaging
2023-03-18T03:22:39.067 INFO:teuthology.lock.ops:Updating [smithi086.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T03:22:39.845 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server
2023-03-18T03:22:39.872 INFO:teuthology.lock.ops:Node 'smithi086.front.sepia.ceph.com' reimaging is complete
2023-03-18T03:22:40.258 INFO:teuthology.lock.ops:Start node 'smithi102.front.sepia.ceph.com' reimaging
2023-03-18T03:22:40.258 INFO:teuthology.lock.ops:Updating [smithi102.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T03:22:40.258 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server
2023-03-18T03:22:40.280 INFO:teuthology.lock.ops:Node 'smithi102.front.sepia.ceph.com' reimaging is complete
2023-03-18T03:22:41.792 INFO:teuthology.lock.ops:Start node 'smithi141.front.sepia.ceph.com' reimaging
2023-03-18T03:22:41.792 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T03:22:41.793 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2023-03-18T03:22:41.822 INFO:teuthology.lock.ops:Node 'smithi141.front.sepia.ceph.com' reimaging is complete
2023-03-18T03:22:44.188 INFO:teuthology.lock.ops:Start node 'smithi159.front.sepia.ceph.com' reimaging
2023-03-18T03:22:44.188 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T03:22:44.188 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server
2023-03-18T03:22:44.213 INFO:teuthology.lock.ops:Node 'smithi159.front.sepia.ceph.com' reimaging is complete
2023-03-18T03:22:46.420 INFO:teuthology.lock.ops:Start node 'smithi181.front.sepia.ceph.com' reimaging
2023-03-18T03:22:46.420 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T03:22:46.420 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2023-03-18T03:22:46.446 INFO:teuthology.lock.ops:Node 'smithi181.front.sepia.ceph.com' reimaging is complete
2023-03-18T03:22:48.883 INFO:teuthology.provision.fog.smithi086:Scheduling deploy of rhel 8.4
2023-03-18T03:22:50.782 INFO:teuthology.provision.fog.smithi102:Scheduling deploy of rhel 8.4
2023-03-18T03:22:52.396 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of rhel 8.4
2023-03-18T03:22:54.369 INFO:teuthology.provision.fog.smithi159:Scheduling deploy of rhel 8.4
2023-03-18T03:22:55.684 INFO:teuthology.provision.fog.smithi181:Scheduling deploy of rhel 8.4
2023-03-18T03:22:57.111 INFO:teuthology.orchestra.console:Power off smithi086
2023-03-18T03:22:58.346 INFO:teuthology.orchestra.console:Power off smithi102
2023-03-18T03:23:09.253 INFO:teuthology.orchestra.console:Power off for smithi086 completed
2023-03-18T03:23:11.446 INFO:teuthology.orchestra.console:Power off smithi159
2023-03-18T03:23:14.301 INFO:teuthology.orchestra.console:Power on smithi086
2023-03-18T03:23:18.319 INFO:teuthology.orchestra.console:Power off smithi141
2023-03-18T03:23:20.739 INFO:teuthology.orchestra.console:Power off smithi181
2023-03-18T03:23:22.487 INFO:teuthology.orchestra.console:Power off for smithi102 completed
2023-03-18T03:23:24.856 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2023-03-18T03:23:26.162 INFO:teuthology.orchestra.console:Power on smithi102
2023-03-18T03:23:27.992 INFO:teuthology.orchestra.console:Power on smithi141
2023-03-18T03:23:29.410 INFO:teuthology.orchestra.console:Power off for smithi159 completed
2023-03-18T03:23:30.615 INFO:teuthology.orchestra.console:Power on for smithi086 completed
2023-03-18T03:23:30.715 INFO:teuthology.orchestra.console:Power on smithi159
2023-03-18T03:23:31.308 INFO:teuthology.provision.fog.smithi086:Waiting for deploy to finish
2023-03-18T03:23:34.947 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2023-03-18T03:23:36.385 INFO:teuthology.orchestra.console:Power on for smithi102 completed
2023-03-18T03:23:36.385 INFO:teuthology.orchestra.console:Power on for smithi141 completed
2023-03-18T03:23:36.486 INFO:teuthology.orchestra.console:Power on smithi181
2023-03-18T03:23:38.213 INFO:teuthology.provision.fog.smithi102:Waiting for deploy to finish
2023-03-18T03:23:40.253 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish
2023-03-18T03:23:42.686 INFO:teuthology.orchestra.console:Power on for smithi181 completed
2023-03-18T03:23:44.626 INFO:teuthology.orchestra.console:Power on for smithi159 completed
2023-03-18T03:23:44.727 INFO:teuthology.provision.fog.smithi181:Waiting for deploy to finish
2023-03-18T03:23:46.723 INFO:teuthology.provision.fog.smithi159:Waiting for deploy to finish
2023-03-18T03:23:49.171 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-03-18T03:23:52.293 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:23:58.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-03-18T03:24:03.429 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:24:07.557 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:24:09.413 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-03-18T03:24:13.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:24:18.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:24:19.941 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-03-18T03:24:24.677 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:24:28.421 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:24:30.661 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-03-18T03:24:34.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:24:39.109 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:24:41.185 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-03-18T03:24:45.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:24:50.245 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:25:05.669 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.141
2023-03-18T03:25:10.277 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:25:13.440 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:16.158 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:17.842 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:25:19.684 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:21.928 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:22.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.159
2023-03-18T03:25:23.638 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:25.684 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:27.311 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:28.436 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:32.903 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:36.406 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:37.292 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:39.018 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:40.017 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:40.084 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:41.578 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:42.518 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:42.585 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:44.101 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:47.006 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:48.625 WARNING:teuthology.provision.fog:'connect to smithi141.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds
2023-03-18T03:25:48.643 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:50.052 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:51.104 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:53.814 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:55.009 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:56.887 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:56.954 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:25:59.350 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T03:25:59.996 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T03:26:01.217 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T03:26:01.217 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T03:26:01.217 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T03:26:01.217 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T03:26:01.217 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T03:26:01.217 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T03:26:01.217 ERROR:paramiko.transport:EOFError
2023-03-18T03:26:01.217 ERROR:paramiko.transport:
2023-03-18T03:26:01.217 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T03:26:01.217 ERROR:paramiko.transport:
2023-03-18T03:26:01.217 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T03:26:01.217 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T03:26:02.253 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T03:26:02.253 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T03:26:02.253 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T03:26:02.253 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T03:26:02.253 ERROR:paramiko.transport:
2023-03-18T03:26:02.255 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T03:26:04.068 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:04.766 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:05.998 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:08.405 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:09.175 WARNING:teuthology.provision.fog:'connect to smithi181.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds
2023-03-18T03:26:09.408 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:10.569 WARNING:teuthology.provision.fog:'connect to smithi159.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds
2023-03-18T03:26:12.228 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:15.237 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86
2023-03-18T03:26:16.795 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:18.187 ERROR:teuthology.orchestra.connection:Error authenticating with smithi159.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:18.801 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:20.212 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T03:26:22.077 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T03:26:22.077 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T03:26:22.078 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T03:26:22.078 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T03:26:22.078 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T03:26:22.078 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T03:26:22.078 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T03:26:22.078 ERROR:paramiko.transport:EOFError
2023-03-18T03:26:22.078 ERROR:paramiko.transport:
2023-03-18T03:26:22.078 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T03:26:22.078 ERROR:paramiko.transport:
2023-03-18T03:26:22.079 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T03:26:22.079 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T03:26:22.079 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T03:26:22.079 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T03:26:22.079 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T03:26:22.079 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T03:26:22.079 ERROR:paramiko.transport:
2023-03-18T03:26:22.079 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T03:26:22.148 ERROR:teuthology.orchestra.connection:Error authenticating with smithi181.front.sepia.ceph.com: Authentication failed.
2023-03-18T03:26:25.189 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102
2023-03-18T03:26:26.309 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86
2023-03-18T03:26:35.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102
2023-03-18T03:26:37.480 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86
2023-03-18T03:26:46.533 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102
2023-03-18T03:26:47.759 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86
2023-03-18T03:26:56.837 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102
2023-03-18T03:26:57.754 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86
2023-03-18T03:27:06.821 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.102
2023-03-18T03:27:09.721 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.86
2023-03-18T03:27:14.057 WARNING:teuthology.provision.fog:No existing session
2023-03-18T03:27:24.258 WARNING:teuthology.provision.fog:timed out
2023-03-18T03:27:28.082 WARNING:teuthology.provision.fog:timed out
2023-03-18T03:27:47.462 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T03:27:50.277 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:28:00.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.181
2023-03-18T03:28:09.615 INFO:teuthology.provision.fog.smithi102:Node is ready
2023-03-18T03:28:16.299 WARNING:teuthology.provision.fog:timed out
2023-03-18T03:28:16.317 INFO:teuthology.orchestra.run.smithi102.stdout:smithi102
2023-03-18T03:28:18.555 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.15.102 smithi102.front.sepia.ceph.com smithi102
2023-03-18T03:28:20.485 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2023-03-18T03:28:23.354 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T03:28:25.036 WARNING:teuthology.provision.fog:timed out
2023-03-18T03:28:25.237 INFO:teuthology.provision.fog.smithi102:Deploy complete!
2023-03-18T03:28:28.853 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T03:28:30.104 INFO:teuthology.provision.fog.smithi181:Node is ready
2023-03-18T03:28:32.132 INFO:teuthology.provision.fog.smithi086:Node is ready
2023-03-18T03:28:33.377 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com
2023-03-18T03:28:35.572 INFO:teuthology.orchestra.run.smithi086.stdout:smithi086
2023-03-18T03:28:39.327 WARNING:teuthology.provision.fog:timed out
2023-03-18T03:28:40.466 INFO:teuthology.orchestra.run.smithi181.stdout:172.21.15.181 smithi181.front.sepia.ceph.com smithi181
2023-03-18T03:28:42.044 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.15.86 smithi086.front.sepia.ceph.com smithi086
2023-03-18T03:28:43.626 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T03:28:43.710 INFO:teuthology.provision.fog.smithi141:Node is ready
2023-03-18T03:28:45.149 INFO:teuthology.orchestra.run.smithi141.stdout:smithi141.front.sepia.ceph.com
2023-03-18T03:28:45.864 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141
2023-03-18T03:28:47.359 INFO:teuthology.provision.fog.smithi181:Deploy complete!
2023-03-18T03:28:48.922 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T03:28:48.950 INFO:teuthology.provision.fog.smithi086:Deploy complete!
2023-03-18T03:28:51.402 INFO:teuthology.provision.fog.smithi159:Node is ready
2023-03-18T03:28:52.310 INFO:teuthology.orchestra.run.smithi159.stdout:smithi159.front.sepia.ceph.com
2023-03-18T03:28:53.753 INFO:teuthology.orchestra.run.smithi159.stdout:172.21.15.159 smithi159.front.sepia.ceph.com smithi159
2023-03-18T03:28:54.891 INFO:teuthology.provision.fog.smithi141:Deploy complete!
2023-03-18T03:28:55.886 INFO:teuthology.provision.fog.smithi159:Deploy complete!
2023-03-18T03:28:56.757 INFO:teuthology.lock.ops:Checking smithi086.front.sepia.ceph.com
2023-03-18T03:28:57.090 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T03:28:57.903 INFO:teuthology.lock.ops:Checking smithi102.front.sepia.ceph.com
2023-03-18T03:28:58.727 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T03:28:58.753 INFO:teuthology.lock.ops:Checking smithi141.front.sepia.ceph.com
2023-03-18T03:29:00.365 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T03:29:00.394 INFO:teuthology.lock.ops:Checking smithi159.front.sepia.ceph.com
2023-03-18T03:29:00.912 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T03:29:00.937 INFO:teuthology.lock.ops:Checking smithi181.front.sepia.ceph.com
2023-03-18T03:29:01.607 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T03:29:01.638 INFO:teuthology.lock.ops:Updating [smithi086.front.sepia.ceph.com]: set os type and version on server
2023-03-18T03:29:03.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T03:29:03.499 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64
2023-03-18T03:29:04.649 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Red Hat Enterprise Linux"
2023-03-18T03:29:05.642 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="8.4 (Ootpa)"
2023-03-18T03:29:05.642 INFO:teuthology.orchestra.run.smithi086.stdout:ID="rhel"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="fedora"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="8.4"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el8"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.redhat.com/"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-18T03:29:05.643 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-18T03:29:05.644 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server
2023-03-18T03:29:05.669 INFO:teuthology.lock.ops:Updating [smithi102.front.sepia.ceph.com]: set os type and version on server
2023-03-18T03:29:06.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T03:29:07.085 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64
2023-03-18T03:29:07.541 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="Red Hat Enterprise Linux"
2023-03-18T03:29:08.201 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="8.4 (Ootpa)"
2023-03-18T03:29:08.202 INFO:teuthology.orchestra.run.smithi102.stdout:ID="rhel"
2023-03-18T03:29:08.538 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="fedora"
2023-03-18T03:29:08.539 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="8.4"
2023-03-18T03:29:08.919 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el8"
2023-03-18T03:29:08.920 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-18T03:29:08.920 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31"
2023-03-18T03:29:08.920 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-18T03:29:09.398 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://www.redhat.com/"
2023-03-18T03:29:12.086 INFO:teuthology.orchestra.run.smithi102.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-18T03:29:12.086 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T03:29:12.086 INFO:teuthology.orchestra.run.smithi102.stdout:
2023-03-18T03:29:12.086 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T03:29:12.086 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-18T03:29:12.086 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-18T03:29:12.086 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-18T03:29:12.087 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server
2023-03-18T03:29:12.118 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: set os type and version on server
2023-03-18T03:29:13.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T03:29:13.684 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64
2023-03-18T03:29:16.336 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Red Hat Enterprise Linux"
2023-03-18T03:29:18.010 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="8.4 (Ootpa)"
2023-03-18T03:29:18.010 INFO:teuthology.orchestra.run.smithi141.stdout:ID="rhel"
2023-03-18T03:29:18.010 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE="fedora"
2023-03-18T03:29:18.010 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="8.4"
2023-03-18T03:29:18.010 INFO:teuthology.orchestra.run.smithi141.stdout:PLATFORM_ID="platform:el8"
2023-03-18T03:29:18.010 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-18T03:29:18.010 INFO:teuthology.orchestra.run.smithi141.stdout:ANSI_COLOR="0;31"
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.redhat.com/"
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-18T03:29:18.993 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-18T03:29:18.994 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2023-03-18T03:29:19.025 INFO:teuthology.lock.ops:Updating [smithi159.front.sepia.ceph.com]: set os type and version on server
2023-03-18T03:29:22.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T03:29:22.720 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64
2023-03-18T03:29:24.368 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="Red Hat Enterprise Linux"
2023-03-18T03:29:27.375 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="8.4 (Ootpa)"
2023-03-18T03:29:27.375 INFO:teuthology.orchestra.run.smithi159.stdout:ID="rhel"
2023-03-18T03:29:27.376 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="fedora"
2023-03-18T03:29:27.376 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="8.4"
2023-03-18T03:29:27.376 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el8"
2023-03-18T03:29:27.376 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-18T03:29:27.376 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31"
2023-03-18T03:29:27.376 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-18T03:29:27.376 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://www.redhat.com/"
2023-03-18T03:29:28.241 INFO:teuthology.orchestra.run.smithi159.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-18T03:29:28.241 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T03:29:28.241 INFO:teuthology.orchestra.run.smithi159.stdout:
2023-03-18T03:29:28.241 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T03:29:28.241 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-18T03:29:28.241 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-18T03:29:28.242 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-18T03:29:28.242 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server
2023-03-18T03:29:28.266 INFO:teuthology.lock.ops:Updating [smithi181.front.sepia.ceph.com]: set os type and version on server
2023-03-18T03:29:29.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T03:29:29.845 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64
2023-03-18T03:29:31.546 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Red Hat Enterprise Linux"
2023-03-18T03:29:33.916 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8.4 (Ootpa)"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:ID="rhel"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="fedora"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8.4"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.redhat.com/"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T03:29:33.917 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2023-03-18T03:29:33.918 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-03-18T03:29:33.918 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2023-03-18T03:29:33.918 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server
2023-03-18T03:29:34.039 INFO:teuthology.dispatcher.supervisor:Running job 7209358
2023-03-18T03:29:35.148 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209358 --name yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi --description rados/cephadm/workunits/{0-distro/rhel_8.4_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon} -- /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209358/orig.config.yaml
2023-03-18T03:29:35.152 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209358
2023-03-18T03:29:37.046 INFO:teuthology.dispatcher.supervisor:Job PID: 13923
2023-03-18T03:29:37.046 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-18T15:30:00.839 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:30:03.311 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:30:04.183 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
KeyError: 'status'
2023-03-18T15:30:05.536 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:30:08.397 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:30:08.397 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:30:08.397 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:30:08.398 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:30:08.398 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:30:08.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:30:08.578 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:30:11.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:30:13.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:30:17.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:30:20.734 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:30:24.695 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-18T15:30:27.644 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-18T15:30:29.050 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-18T15:30:29.050 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-18T15:30:29.689 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:30:36.854 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-mon.smithi181.log: file size changed while zipping
2023-03-18T15:30:39.852 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-03-18T15:30:40.955 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-18T15:30:40.955 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-18T15:30:47.615 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-18T15:30:50.740 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-03-18T15:30:51.896 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.13.log: file size changed while zipping
2023-03-18T15:30:54.621 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.9.log: file size changed while zipping
2023-03-18T15:30:55.946 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-18T15:30:58.473 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-mon.smithi159.log: file size changed while zipping
2023-03-18T15:30:59.959 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.12.log: file size changed while zipping
2023-03-18T15:31:03.104 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2023-03-18T15:31:03.909 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-osd.14.log: file size changed while zipping
2023-03-18T15:31:30.402 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:31:33.151 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:33:34.907 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:33:35.347 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:33:38.434 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:33:40.159 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:33:41.687 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:33:41.687 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:33:41.687 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:33:41.687 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:33:41.687 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:33:41.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:33:41.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:33:43.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:33:44.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:33:45.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:33:46.682 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:33:48.205 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:33:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:33:50.418 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:33:50.421 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:33:52.080 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:33:52.528 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:33:52.528 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:33:52.528 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:33:52.528 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:33:52.528 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:33:52.574 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:33:54.587 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:35:56.444 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:35:58.875 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:36:01.543 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:36:04.585 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:36:06.010 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:36:06.010 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:36:06.010 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:36:06.011 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:36:06.011 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:36:06.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:36:06.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:36:07.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:36:09.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:36:11.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:36:13.008 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:36:15.000 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:36:16.755 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:36:18.800 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:36:18.804 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:36:18.805 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:36:21.657 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:36:21.657 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:36:21.657 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:36:21.658 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:36:21.658 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:36:21.701 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:36:23.315 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:38:24.980 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:38:25.756 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:38:26.769 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:38:28.926 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:38:30.359 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:38:30.359 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:38:30.359 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:38:30.359 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:38:32.224 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:38:32.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:38:32.403 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:38:33.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:38:34.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:38:35.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:38:36.804 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:38:38.040 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:38:43.361 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:38:45.050 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:38:45.054 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:38:47.863 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:38:47.863 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:38:47.863 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:38:47.863 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:38:47.863 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:38:47.863 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:38:51.273 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:38:52.211 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:40:53.199 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:40:54.194 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:40:56.021 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:40:57.244 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:40:58.325 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:40:58.325 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:40:58.325 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:40:58.325 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:40:58.326 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:40:58.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:40:58.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:40:59.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:41:00.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:41:03.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:41:04.459 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:41:07.063 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:41:09.344 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:41:10.574 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:41:10.579 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:41:11.536 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:41:12.772 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:41:12.772 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:41:12.772 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:41:12.772 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:41:12.773 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:41:12.814 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:41:14.278 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:43:14.842 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:43:17.851 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:43:20.183 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:43:22.446 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:43:23.711 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:43:23.711 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:43:23.711 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:43:23.711 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:43:23.711 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:43:23.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:43:23.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:43:25.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:43:29.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:43:30.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:43:32.116 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:43:34.638 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:43:36.562 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:43:38.149 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:43:38.153 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:43:38.154 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:43:38.154 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:43:38.154 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:43:38.154 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:43:38.154 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:43:38.154 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:43:38.238 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:43:40.747 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:45:41.567 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:45:43.575 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:45:45.391 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:45:47.845 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:45:51.989 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:45:51.989 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:45:51.989 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:45:51.989 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:45:51.989 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:45:51.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:45:52.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:45:53.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:45:54.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:45:56.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:45:59.823 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:46:03.184 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:46:03.785 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:46:05.435 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:46:05.439 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:46:06.760 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:46:06.760 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:46:06.761 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:46:06.761 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:46:06.761 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:46:06.761 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:46:06.803 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:46:08.302 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:48:11.895 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:48:13.204 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:48:14.569 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:48:15.719 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:48:16.792 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:48:16.792 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:48:16.792 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:48:16.792 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:48:16.792 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:48:16.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:48:16.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:48:19.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:48:20.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:48:21.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:48:25.700 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:48:27.880 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:48:30.279 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:48:32.518 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:48:32.522 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:48:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:48:36.362 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:48:36.362 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:48:36.362 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:48:38.598 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:48:38.598 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:48:38.641 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:48:39.819 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:50:41.395 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:50:43.896 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:50:45.190 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:50:45.760 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:50:47.336 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:50:47.337 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:50:47.337 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:50:47.337 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:50:47.337 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:50:47.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:50:48.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:50:49.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:50:51.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:50:52.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:50:54.999 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:50:56.475 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:50:58.234 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:51:00.159 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:51:00.162 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:51:00.163 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:51:00.163 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:51:00.163 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:51:00.163 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:51:00.163 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:51:00.163 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:51:00.202 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:51:02.625 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:53:05.350 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:53:06.459 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:53:08.717 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:53:12.036 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:53:13.645 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:53:13.645 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:53:13.645 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:53:13.645 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:53:13.645 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:53:13.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:53:13.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:53:15.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:53:16.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:53:17.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:53:17.978 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:53:19.547 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:53:20.485 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:53:20.862 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:53:20.867 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:53:21.885 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:53:21.885 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:53:22.909 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:53:22.910 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:53:22.910 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:53:23.525 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:53:23.570 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:53:24.810 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:55:25.720 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:55:28.500 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:55:30.761 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:55:33.805 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:55:35.343 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:55:35.344 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:55:35.344 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:55:35.344 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:55:35.344 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:55:35.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:55:35.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:55:37.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:55:39.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:55:40.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:55:42.972 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:55:45.252 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:55:46.419 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:55:50.072 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:55:50.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:55:50.076 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:55:50.076 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:55:50.076 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:55:50.077 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:55:50.077 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:55:50.077 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:55:50.115 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:55:52.089 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:57:53.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T15:57:56.087 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:57:58.242 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T15:57:59.205 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T15:58:00.718 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T15:58:00.718 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T15:58:00.719 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T15:58:00.719 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T15:58:00.719 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:58:00.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T15:58:00.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T15:58:02.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T15:58:08.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T15:58:09.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T15:58:14.454 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:58:17.664 INFO:teuthology.misc:Compressing logs...
2023-03-18T15:58:19.178 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:58:21.890 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:58:21.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T15:58:21.894 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:58:21.895 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:58:21.895 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:58:23.631 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:58:23.631 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T15:58:23.632 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T15:58:23.692 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T15:58:25.385 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:00:28.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:00:29.509 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:00:30.306 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:00:31.954 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:00:32.777 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:00:32.777 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:00:32.777 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:00:32.777 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:00:32.777 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:00:32.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:00:32.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:00:34.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:00:35.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:00:37.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:00:39.280 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:00:40.759 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:00:41.126 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:00:43.107 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:00:43.114 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:00:45.898 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:00:45.898 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:00:45.899 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:00:45.899 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:00:45.899 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:00:45.899 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:00:45.939 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:00:47.544 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:02:52.594 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:02:54.327 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:02:56.158 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:02:57.405 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:02:59.050 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:02:59.050 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:02:59.050 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:02:59.050 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:02:59.051 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:02:59.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:02:59.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:03:01.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:03:02.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:03:04.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:03:06.602 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:03:09.429 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:03:11.339 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:03:15.920 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:03:15.924 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:03:17.832 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:03:17.832 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:03:17.832 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:03:17.832 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:03:17.832 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:03:17.832 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:03:17.872 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:03:21.123 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:05:23.358 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:05:24.500 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:05:26.320 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:05:28.048 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:05:30.065 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:05:30.065 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:05:30.065 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:05:30.065 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:05:31.326 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:05:31.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:05:31.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:05:32.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:05:34.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:05:35.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:05:37.463 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:05:38.888 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:05:39.748 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:05:40.852 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:05:40.856 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:05:42.215 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:05:42.215 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:05:42.215 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:05:42.216 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:05:42.216 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:05:42.216 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:05:42.256 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:05:43.076 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:07:44.574 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:07:46.815 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:07:48.582 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:07:50.459 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:07:51.560 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:07:51.560 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:07:51.560 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:07:51.561 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:07:51.561 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:07:51.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:07:51.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:07:54.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:07:55.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:07:56.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:07:58.191 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:08:00.793 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:08:01.733 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:08:03.536 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:08:03.542 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:08:05.184 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:08:05.185 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:08:05.185 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:08:05.185 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:08:05.185 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:08:05.926 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:08:05.975 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:08:07.068 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:10:09.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:10:12.216 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:10:14.460 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:10:16.900 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:10:18.322 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:10:18.322 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:10:18.322 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:10:18.322 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:10:18.322 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:10:18.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:10:18.548 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:10:20.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:10:21.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:10:22.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:10:23.835 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:10:25.196 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:10:26.663 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:10:28.300 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:10:28.304 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:10:28.305 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:10:28.305 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:10:28.305 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:10:28.305 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:10:28.306 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:10:28.306 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:10:28.343 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:10:29.807 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:12:31.803 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:12:34.137 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:12:35.613 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:12:37.756 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:12:39.247 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:12:39.247 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:12:39.247 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:12:39.247 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:12:39.247 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:12:40.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:12:40.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:12:42.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:12:44.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:12:45.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:12:47.677 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:12:49.202 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:12:50.587 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:12:51.464 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:12:51.470 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:12:52.809 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:12:52.809 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:12:52.809 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:12:52.809 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:12:52.809 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:12:52.809 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:12:52.891 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:12:53.803 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:14:54.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:14:56.126 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:14:57.928 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:14:59.589 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:15:01.185 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:15:01.185 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:15:01.185 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:15:01.185 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:15:01.185 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:15:01.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:15:01.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:15:02.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:15:03.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:15:05.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:15:08.064 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:15:11.358 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:15:14.343 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:15:16.719 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:15:16.726 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:15:18.642 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:15:18.642 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:15:18.642 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:15:20.211 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:15:21.641 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:15:21.641 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:15:21.682 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:15:24.572 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:17:28.266 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:17:32.948 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:17:36.660 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:17:45.007 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:17:46.705 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:17:46.705 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:17:46.705 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:17:48.304 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:17:48.305 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:17:48.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:17:48.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:17:49.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:17:51.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:17:52.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:17:54.452 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:17:56.752 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:17:58.327 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:17:59.318 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:17:59.325 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:18:00.057 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:18:00.996 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:18:01.875 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:18:01.875 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:18:02.851 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:18:02.851 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:18:06.476 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:18:08.817 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:20:10.277 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:20:18.514 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:20:20.229 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:20:21.783 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:20:23.487 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:20:23.487 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:20:23.487 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:20:23.487 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:20:23.488 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:20:23.488 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:20:23.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:20:31.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:20:33.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:20:35.373 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:20:41.970 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:20:43.203 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:20:44.694 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:20:46.527 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:20:46.531 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:20:48.558 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:20:48.558 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:20:48.558 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:20:48.558 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:20:48.558 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:20:48.558 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:20:48.598 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:20:49.652 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:22:51.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:22:52.035 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:22:53.283 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:22:55.331 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:22:56.573 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:22:56.573 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:22:56.573 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:22:56.573 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:22:56.574 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:22:56.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:22:57.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:23:01.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:23:03.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:23:07.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:23:10.083 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:23:11.945 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:23:14.317 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:23:16.260 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:23:16.266 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:23:19.231 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:23:19.231 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:23:19.231 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:23:19.231 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:23:19.231 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:23:19.232 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:23:19.283 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:23:21.111 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:25:22.678 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:25:25.499 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:25:25.991 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:25:26.507 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:25:27.060 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:25:27.060 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:25:27.060 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:25:27.060 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:25:27.060 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:25:27.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:25:27.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:25:27.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:25:28.468 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:25:29.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:25:30.071 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:25:31.790 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:25:32.626 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:25:34.504 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:25:34.508 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:25:36.032 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:25:37.183 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:25:37.183 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:25:37.184 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:25:37.184 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:25:37.184 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:25:37.762 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:25:38.796 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:27:40.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:27:41.320 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:27:42.747 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:27:43.994 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:27:44.857 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:27:44.857 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:27:44.858 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:27:44.859 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:27:44.859 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:27:44.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:27:45.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:27:45.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:27:47.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:27:48.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:27:49.507 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:27:50.830 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:27:52.051 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:27:52.862 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:27:52.865 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:27:54.679 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:27:54.679 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:27:55.633 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:27:55.633 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:27:55.633 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:27:55.633 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:27:56.281 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:27:57.280 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:29:58.026 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:29:58.888 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:30:00.026 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:30:01.421 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:30:03.025 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:30:03.025 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:30:03.025 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:30:03.025 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:30:03.025 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:30:03.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:30:03.212 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:30:04.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:30:06.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:30:09.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:30:11.051 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:30:12.932 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:30:15.154 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:30:17.783 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:30:17.787 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:30:17.787 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:30:17.788 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:30:17.788 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:30:17.788 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:30:17.788 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:30:17.788 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:30:19.286 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:30:21.760 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:32:23.414 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:32:23.452 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:32:24.915 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:32:29.194 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:32:30.956 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:32:30.956 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:32:30.956 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:32:30.956 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:32:32.456 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:32:32.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:32:32.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:32:34.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:32:35.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:32:38.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:32:40.140 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:32:42.914 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:32:45.282 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:32:47.298 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:32:47.302 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:32:47.302 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:32:47.302 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:32:47.302 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:32:47.302 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:32:47.302 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:32:47.302 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:32:49.702 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:32:50.837 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:34:51.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:34:52.719 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:34:53.227 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:34:54.005 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:34:55.156 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:34:55.156 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:34:55.156 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:34:55.156 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:34:55.156 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:34:55.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:34:56.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:35:00.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:35:01.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:35:03.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:35:04.767 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:35:06.223 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:35:07.139 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:35:07.916 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:35:07.921 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:35:09.276 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:35:10.681 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:35:10.682 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:35:10.683 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:35:10.683 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:35:10.683 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:35:10.723 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:35:12.800 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:37:15.417 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:37:17.094 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:37:18.095 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:37:19.192 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:37:20.245 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:37:20.245 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:37:20.245 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:37:20.245 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:37:20.245 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:37:20.245 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:37:20.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:37:22.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:37:24.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:37:27.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:37:29.560 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:37:33.206 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:37:36.235 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:37:38.571 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:37:38.575 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:37:39.751 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:37:39.752 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:37:39.752 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:37:39.752 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:37:39.752 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:37:39.752 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:37:39.800 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:37:41.706 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:39:44.593 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:39:45.437 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:39:46.944 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:39:49.028 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:39:54.249 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:39:54.249 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:39:54.249 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:39:54.249 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:39:54.249 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:39:54.250 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:39:54.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:39:55.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:39:58.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:40:02.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:40:05.361 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:40:09.328 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:40:14.254 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:40:16.856 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:40:16.861 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:40:18.495 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:40:20.717 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:40:20.718 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:40:20.718 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:40:20.718 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:40:20.718 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:40:20.760 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:40:24.259 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:42:25.936 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:42:27.540 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:42:28.723 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:42:30.696 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:42:32.814 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:42:32.814 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:42:32.814 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:42:32.814 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:42:32.814 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:42:33.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:42:33.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:42:35.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:42:41.221 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:42:42.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:42:45.707 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:42:48.939 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:42:50.392 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:42:52.165 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:42:52.169 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:42:54.823 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:42:57.057 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:42:57.057 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:42:57.057 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:42:57.057 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:42:57.057 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:42:57.148 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:42:59.963 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:45:01.794 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:45:04.808 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:45:07.525 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:45:09.716 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:45:11.429 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:45:11.429 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:45:11.429 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:45:11.429 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:45:14.191 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:45:14.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:45:14.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:45:15.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:45:16.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:45:19.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:45:21.060 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:45:23.934 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:45:25.017 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:45:26.129 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:45:26.134 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:45:28.628 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:45:28.628 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:45:28.629 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:45:28.629 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:45:28.629 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:45:28.629 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:45:28.669 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:45:32.729 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:47:33.929 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:47:34.550 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:47:35.150 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:47:36.849 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:47:38.452 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:47:38.452 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:47:38.452 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:47:38.452 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:47:38.452 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:47:38.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:47:38.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:47:43.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:47:45.140 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:47:49.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:47:51.499 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:47:53.448 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:47:54.419 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:47:56.866 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:47:56.871 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:47:58.712 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:48:00.338 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:48:01.144 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:48:01.144 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:48:01.144 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:48:01.145 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:48:01.185 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:48:06.112 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:50:08.279 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:50:09.640 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:50:10.879 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:50:12.827 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:50:15.285 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:50:15.285 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:50:15.285 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:50:15.285 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:50:15.285 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:50:15.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:50:15.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:50:17.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:50:21.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:50:23.265 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:50:25.857 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:50:27.690 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:50:29.479 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:50:31.290 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:50:31.908 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:50:33.263 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:50:33.812 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:50:33.812 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:50:33.812 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:50:33.813 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:50:33.813 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:50:33.851 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:50:35.706 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:52:37.247 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:52:38.090 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:52:40.361 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:52:41.770 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:52:42.120 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:52:42.120 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:52:42.120 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:52:42.120 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:52:42.120 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:52:42.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:52:42.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:52:42.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:52:43.653 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:52:45.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:52:47.637 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:52:49.820 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:52:50.996 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:52:51.469 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:52:51.472 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:52:54.113 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:52:54.113 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:52:54.113 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:52:54.113 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:52:54.113 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:52:54.113 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:52:54.152 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:52:55.306 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:54:56.226 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:54:57.425 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:54:59.641 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:55:01.507 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:55:03.055 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:55:03.055 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:55:03.055 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:55:03.055 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:55:04.801 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:55:05.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:55:07.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:55:08.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:55:10.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:55:12.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:55:15.925 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:55:18.057 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:55:19.411 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:55:21.454 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:55:21.458 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:55:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:55:24.929 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:55:24.929 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:55:24.929 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:55:24.930 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:55:24.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:55:24.992 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:55:27.466 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:57:29.042 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:57:30.217 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:57:31.320 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:57:32.279 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T16:57:33.803 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T16:57:33.803 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T16:57:33.803 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T16:57:33.803 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T16:57:33.803 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:57:33.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T16:57:33.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T16:57:34.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T16:57:37.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T16:57:37.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T16:57:38.804 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:57:40.466 INFO:teuthology.misc:Compressing logs...
2023-03-18T16:57:41.889 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:57:44.623 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:57:44.626 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T16:57:44.627 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:57:44.627 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:57:44.627 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:57:47.013 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:57:47.013 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T16:57:47.013 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T16:57:47.055 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:57:49.209 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:59:50.753 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T16:59:53.178 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T16:59:56.109 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T16:59:59.881 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:00:01.862 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:00:01.862 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:00:01.862 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:00:01.862 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:00:01.862 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:00:01.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:00:05.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:00:07.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:00:10.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:00:12.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:00:16.397 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:00:18.948 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:00:19.789 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:00:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:00:21.984 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:00:25.494 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:00:25.494 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:00:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:00:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:00:25.495 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:00:25.495 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:00:25.542 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:00:27.683 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:02:30.598 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:02:31.695 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:02:33.309 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:02:35.109 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:02:37.011 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:02:37.011 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:02:37.012 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:02:37.014 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:02:39.843 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:02:39.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:02:40.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:02:43.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:02:46.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:02:48.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:02:54.684 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:02:57.082 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:02:58.436 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:02:59.460 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:02:59.464 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:03:00.250 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:03:00.250 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:03:00.250 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:03:00.250 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:03:00.251 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:03:00.251 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:03:00.291 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:03:02.495 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:05:03.056 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:05:04.753 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:05:07.228 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:05:10.376 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:05:12.415 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:05:12.415 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:05:12.415 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:05:12.415 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:05:12.415 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:05:12.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:05:12.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:05:13.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:05:14.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:05:15.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:05:17.317 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:05:20.422 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:05:22.113 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:05:23.888 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:05:23.893 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:05:25.478 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:05:25.478 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:05:25.479 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:05:27.906 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:05:27.906 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:05:27.906 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:05:27.945 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:05:29.936 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:07:32.966 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:07:33.772 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:07:35.468 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:07:36.748 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:07:37.882 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:07:37.883 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:07:37.883 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:07:37.883 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:07:37.883 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:07:37.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:07:38.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:07:40.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:07:44.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:07:46.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:07:50.006 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:07:52.190 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:07:53.610 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:07:54.282 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:07:54.286 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:07:56.820 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:07:56.820 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:07:56.820 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:07:56.820 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:07:56.820 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:07:56.821 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:07:56.870 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:07:57.704 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:09:59.204 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:10:01.372 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:10:03.064 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:10:05.685 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:10:07.594 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:10:07.594 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:10:10.735 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:10:10.736 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:10:10.736 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:10:10.736 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:10:11.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:10:14.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:10:15.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:10:17.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:10:19.576 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:10:21.136 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:10:21.909 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:10:22.901 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:10:22.906 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:10:24.129 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:10:24.129 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:10:24.129 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:10:24.129 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:10:24.130 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:10:24.130 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:10:24.167 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:10:25.776 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:12:27.163 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:12:27.406 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:12:28.953 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:12:29.981 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:12:32.078 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:12:32.869 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:12:32.870 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:12:32.870 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:12:32.870 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:12:35.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:12:35.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:12:37.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:12:39.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:12:40.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:12:47.396 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:12:51.447 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:12:53.771 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:12:56.495 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:12:56.500 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:12:58.303 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:12:58.825 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:12:58.826 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:12:58.826 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:12:58.826 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:12:58.826 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:12:58.912 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:13:02.653 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:15:04.831 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:15:07.996 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:15:09.594 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:15:11.964 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:15:13.933 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:15:13.933 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:15:13.933 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:15:13.933 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:15:13.933 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:15:13.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:15:14.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:15:16.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:15:18.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:15:19.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:15:21.766 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:15:23.391 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:15:24.050 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:15:25.256 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:15:25.262 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:15:26.479 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:15:26.480 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:15:26.480 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:15:26.480 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:15:26.480 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:15:26.480 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:15:26.523 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:15:27.246 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:17:27.749 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:17:28.419 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:17:29.674 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:17:31.154 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:17:31.814 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:17:31.815 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:17:31.815 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:17:31.815 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:17:31.815 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:17:31.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:17:31.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:17:32.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:17:34.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:17:36.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:17:38.416 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:17:43.683 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:17:45.464 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:17:48.673 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:17:48.678 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:17:50.808 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:17:50.808 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:17:50.808 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:17:50.809 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:17:52.167 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:17:53.565 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:17:53.605 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:17:55.455 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:19:58.367 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:19:59.631 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:20:00.747 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:20:01.780 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:20:03.942 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:20:03.942 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:20:03.942 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:20:03.942 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:20:03.943 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:20:03.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:20:04.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:20:05.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:20:06.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:20:08.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:20:09.984 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:20:11.234 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:20:11.730 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:20:12.787 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:20:12.792 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:20:12.793 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:20:13.476 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:20:14.148 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:20:14.663 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:20:14.664 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:20:14.664 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:20:14.705 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:20:15.380 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:22:16.371 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:22:17.289 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:22:18.897 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:22:20.294 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:22:21.129 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:22:21.129 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:22:21.129 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:22:22.144 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:22:22.144 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:22:22.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:22:22.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:22:24.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:22:25.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:22:26.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:22:28.223 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:22:30.370 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:22:31.947 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:22:34.175 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:22:34.179 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:22:34.179 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:22:34.179 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:22:34.180 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:22:34.180 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:22:34.180 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:22:34.180 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:22:34.219 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:22:35.432 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:24:37.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:24:38.878 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:24:40.211 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:24:41.524 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:24:42.548 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:24:42.548 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:24:42.548 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:24:42.548 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:24:42.548 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:24:42.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:24:42.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:24:45.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:24:46.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:24:47.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:24:50.003 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:24:53.891 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:24:55.254 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:24:58.517 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:24:59.260 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:25:01.679 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:25:01.679 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:25:01.679 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:25:01.679 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:25:01.680 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:25:01.680 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:25:01.719 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:25:04.601 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:27:06.731 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:27:08.416 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:27:09.838 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:27:11.317 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:27:12.499 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:27:13.926 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:27:13.926 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:27:13.926 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:27:13.926 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:27:14.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:27:16.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:27:17.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:27:18.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:27:22.217 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:27:26.585 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:27:28.721 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:27:30.745 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:27:31.516 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:27:31.519 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:27:32.869 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:27:32.869 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:27:32.869 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:27:32.869 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:27:32.869 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:27:33.312 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:27:33.351 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:27:34.373 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:29:34.695 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:29:38.214 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:29:42.317 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:29:46.881 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:29:52.050 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:29:52.050 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:29:52.050 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:29:52.050 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:29:52.051 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:29:52.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:29:52.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:29:55.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:29:57.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:29:59.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:30:01.150 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:30:03.241 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:30:05.685 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:30:06.631 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:30:06.634 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:30:07.952 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:30:07.952 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:30:07.952 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:30:07.952 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:30:07.952 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:30:07.952 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:30:07.992 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:30:09.541 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:32:11.249 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:32:13.032 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:32:14.805 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:32:16.726 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:32:17.744 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:32:17.744 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:32:17.744 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:32:17.744 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:32:17.744 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:32:18.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:32:18.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:32:19.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:32:21.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:32:24.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:32:26.413 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:32:29.496 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:32:30.973 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:32:33.762 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:32:33.767 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:32:34.995 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:32:34.995 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:32:34.995 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:32:34.995 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:32:34.996 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:32:36.670 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:32:36.709 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:32:37.797 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:34:39.425 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:34:40.444 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:34:42.136 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:34:44.742 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:34:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:34:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:34:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:34:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:34:47.521 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:34:48.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:34:48.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:34:49.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:34:50.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:34:51.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:34:53.447 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:34:55.495 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:34:57.097 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:34:59.425 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:34:59.432 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:35:00.706 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:35:01.234 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:35:01.234 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:35:01.234 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:35:01.234 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:35:02.496 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:35:02.538 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:35:03.739 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:37:05.571 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:37:06.239 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:37:08.051 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:37:08.871 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:37:09.867 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:37:09.867 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:37:09.867 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:37:09.867 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:37:09.867 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:37:09.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:37:10.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:37:10.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:37:11.485 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:37:13.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:37:14.784 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:37:16.965 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:37:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:37:23.295 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:37:23.301 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:37:25.030 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:37:25.031 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:37:25.031 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:37:25.031 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:37:25.031 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:37:25.031 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:37:25.088 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:37:26.907 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:39:27.606 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:39:29.061 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:39:31.319 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:39:32.879 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:39:34.922 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:39:34.922 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:39:34.922 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:39:34.922 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:39:34.922 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:39:34.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:39:35.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:39:35.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:39:36.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:39:38.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:39:39.246 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:39:41.554 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:39:43.283 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:39:44.525 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:39:44.532 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:39:46.790 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:39:46.790 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:39:46.791 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:39:46.791 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:39:46.791 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:39:46.791 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:39:46.837 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:39:50.157 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:41:53.277 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:41:55.030 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:41:56.424 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:41:58.282 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:42:01.265 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:42:01.265 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:42:01.265 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:42:01.265 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:42:01.266 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:42:01.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:42:01.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:42:02.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:42:04.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:42:05.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:42:06.588 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:42:08.153 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:42:10.414 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:42:12.164 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:42:12.169 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:42:14.115 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:42:15.611 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:42:15.612 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:42:15.612 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:42:15.612 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:42:15.612 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:42:15.699 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:42:17.179 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:44:19.211 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:44:21.481 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:44:23.044 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:44:24.584 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:44:26.197 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:44:26.197 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:44:26.197 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:44:26.197 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:44:26.197 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:44:26.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:44:26.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:44:27.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:44:28.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:44:29.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:44:30.518 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:44:31.801 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:44:32.119 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:44:32.832 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:44:32.836 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:44:32.836 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:44:32.837 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:44:32.837 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:44:32.837 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:44:32.837 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:44:32.837 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:44:32.875 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:44:33.717 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:46:34.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:46:36.058 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:46:40.439 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:46:42.997 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:46:44.948 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:46:44.948 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:46:44.948 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:46:44.948 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:46:44.948 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:46:44.948 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:46:45.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:46:46.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:46:47.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:46:48.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:46:50.297 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:46:52.471 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:46:54.968 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:46:56.416 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:46:56.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:46:57.297 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:46:57.297 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:46:57.297 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:46:57.297 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:46:57.300 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:46:57.300 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:46:57.342 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:46:58.421 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:48:59.543 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:49:02.242 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:49:05.300 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:49:07.990 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:49:09.492 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:49:11.209 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:49:11.209 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:49:11.210 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:49:11.210 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:49:11.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:49:11.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:49:12.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:49:13.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:49:15.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:49:19.161 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:49:25.628 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:49:26.586 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:49:28.963 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:49:28.967 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:49:30.898 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:49:32.576 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:49:32.576 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:49:32.576 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:49:32.576 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:49:32.576 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:49:32.618 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:49:33.649 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:51:34.669 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:51:36.341 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:51:38.482 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:51:39.743 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:51:42.193 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:51:42.193 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:51:42.193 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:51:42.193 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:51:42.193 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:51:42.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:51:42.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:51:43.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:51:45.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:51:46.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:51:47.729 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:51:49.337 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:51:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:51:51.524 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:51:51.528 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:51:53.030 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:51:53.030 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:51:53.030 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:51:53.030 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:51:53.030 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:51:53.030 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:51:53.073 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:51:54.737 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:53:55.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:53:58.535 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:53:59.685 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:54:00.443 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:54:02.311 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:54:02.312 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:54:02.312 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:54:02.312 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:54:02.312 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:54:02.312 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:54:02.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:54:03.328 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:54:04.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:54:07.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:54:09.811 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:54:11.312 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:54:12.082 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:54:13.121 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:54:13.125 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:54:14.491 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:54:14.491 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:54:14.491 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:54:14.491 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:54:14.492 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:54:14.492 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:54:14.540 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:54:15.414 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:56:16.783 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:56:18.724 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:56:20.727 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:56:22.658 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:56:25.604 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:56:25.604 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:56:25.604 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:56:25.604 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:56:25.604 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:56:25.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:56:28.435 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:56:30.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:56:32.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:56:33.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:56:34.573 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:56:35.739 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:56:36.644 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:56:37.904 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:56:37.908 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:56:39.082 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:56:39.082 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:56:39.082 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:56:39.083 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:56:39.083 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:56:39.083 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:56:39.124 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:56:41.622 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:58:44.653 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T17:58:46.761 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:58:48.440 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T17:58:51.229 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T17:58:52.792 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T17:58:54.763 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T17:58:54.763 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T17:58:54.764 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T17:58:54.764 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:58:54.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T17:58:54.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T17:58:57.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T17:58:59.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T17:59:00.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T17:59:01.870 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:59:03.810 INFO:teuthology.misc:Compressing logs...
2023-03-18T17:59:04.923 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:59:06.421 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:59:06.424 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T17:59:07.924 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:59:07.924 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:59:07.924 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:59:09.037 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:59:09.037 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T17:59:09.037 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T17:59:09.076 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T17:59:09.514 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:01:11.270 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:01:13.065 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:01:15.212 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:01:17.051 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:01:18.272 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:01:18.272 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:01:18.272 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:01:18.272 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:01:18.272 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:01:18.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:01:18.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:01:19.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:01:20.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:01:21.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:01:23.524 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:01:24.901 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:01:25.654 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:01:26.414 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:01:26.418 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:01:26.808 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:01:26.808 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:01:26.808 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:01:26.808 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:01:26.808 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:01:26.808 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:01:26.847 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:01:27.450 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:03:28.340 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:03:29.173 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:03:30.167 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:03:32.267 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:03:36.756 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:03:36.757 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:03:36.757 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:03:36.757 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:03:36.757 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:03:39.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:03:39.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:03:42.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:03:44.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:03:48.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:03:50.791 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:03:53.818 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:03:55.044 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:03:56.061 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:03:56.066 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:03:56.914 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:03:58.095 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:03:58.096 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:03:58.096 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:03:58.096 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:03:58.096 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:03:58.134 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:04:00.488 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:06:02.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:06:05.023 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:06:07.897 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:06:09.605 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:06:11.945 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:06:11.945 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:06:11.945 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:06:11.945 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:06:11.945 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:06:11.945 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:06:12.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:06:14.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:06:16.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:06:17.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:06:19.318 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:06:20.223 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:06:21.481 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:06:26.250 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:06:26.254 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:06:26.254 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:06:26.255 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:06:28.610 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:06:28.611 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:06:28.611 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:06:28.611 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:06:28.651 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:06:31.008 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:08:32.484 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:08:33.652 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:08:34.946 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:08:36.035 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:08:37.828 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:08:37.828 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:08:37.828 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:08:37.828 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:08:37.828 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:08:37.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:08:38.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:08:38.813 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:08:40.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:08:41.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:08:43.653 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:08:46.697 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:08:47.574 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:08:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:08:49.329 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:08:49.909 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:08:49.909 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:08:49.910 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:08:49.910 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:08:49.910 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:08:49.910 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:08:49.950 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:08:50.289 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:10:51.353 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:10:52.542 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:10:53.152 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:10:55.165 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:10:55.694 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:10:55.695 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:10:56.518 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:10:56.518 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:10:56.518 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:10:56.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:10:56.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:10:57.841 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:10:58.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:11:01.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:11:03.210 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:11:05.361 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:11:06.562 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:11:07.985 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:11:07.989 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:11:10.056 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:11:10.056 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:11:10.056 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:11:10.056 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:11:10.056 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:11:10.057 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:11:10.139 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:11:11.861 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:13:14.199 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:13:17.408 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:13:21.012 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:13:22.983 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:13:24.623 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:13:24.623 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:13:24.623 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:13:24.623 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:13:24.623 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:13:24.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:13:24.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:13:27.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:13:29.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:13:32.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:13:34.362 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:13:36.008 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:13:36.347 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:13:37.825 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:13:37.829 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:13:39.903 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:13:39.903 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:13:40.907 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:13:40.907 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:13:40.908 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:13:42.918 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:13:42.957 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:13:45.007 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:15:46.750 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:15:49.268 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:15:50.648 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:15:52.143 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:15:55.891 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:15:55.891 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:15:55.891 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:15:55.891 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:15:55.891 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:15:55.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:15:56.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:15:59.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:16:01.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:16:03.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:16:04.262 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:16:06.674 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:16:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:16:08.951 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:16:09.163 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:16:10.185 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:16:10.995 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:16:10.995 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:16:10.995 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:16:10.995 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:16:10.995 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:16:11.035 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:16:12.760 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:18:13.893 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:18:18.361 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:18:19.795 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:18:21.447 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:18:23.265 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:18:23.265 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:18:23.266 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:18:23.266 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:18:23.266 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:18:23.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:18:23.443 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:18:24.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:18:25.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:18:25.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:18:27.389 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:18:30.476 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:18:33.441 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:18:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:18:35.175 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:18:37.011 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:18:37.011 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:18:37.011 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:18:37.011 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:18:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:18:37.011 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:18:37.049 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:18:38.459 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:20:40.178 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:20:42.195 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:20:44.278 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:20:45.217 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:20:46.344 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:20:46.344 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:20:46.344 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:20:46.345 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:20:46.345 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:20:46.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:20:46.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:20:48.213 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:20:49.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:20:51.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:20:54.971 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:20:58.012 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:20:59.407 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:21:00.788 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:21:00.793 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:21:02.384 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:21:02.384 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:21:02.384 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:21:02.384 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:21:02.384 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:21:02.384 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:21:02.426 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:21:04.814 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:23:08.090 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:23:09.581 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:23:11.989 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:23:14.977 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:23:16.437 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:23:16.438 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:23:16.438 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:23:16.438 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:23:16.438 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:23:16.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:23:16.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:23:21.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:23:23.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:23:25.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:23:27.303 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:23:28.911 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:23:30.698 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:23:31.813 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:23:31.816 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:23:32.920 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:23:32.920 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:23:33.372 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:23:33.372 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:23:34.869 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:23:34.870 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:23:34.911 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:23:38.279 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:25:40.240 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:25:41.245 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:25:42.215 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:25:44.058 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:25:46.067 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:25:46.067 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:25:46.067 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:25:46.068 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:25:46.068 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:25:46.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:25:46.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:25:51.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:25:53.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:25:56.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:25:59.421 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:26:01.290 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:26:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:26:05.224 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:26:05.227 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:26:06.032 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:26:06.032 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:26:06.032 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:26:06.032 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:26:06.032 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:26:06.032 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:26:06.074 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:26:08.194 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:28:14.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:28:15.354 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:28:17.464 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:28:18.577 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:28:20.793 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:28:20.794 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:28:20.794 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:28:20.794 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:28:20.794 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:28:20.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:28:20.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:28:23.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:28:26.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:28:28.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:28:31.135 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:28:32.381 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:28:33.608 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:28:34.704 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:28:34.708 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:28:37.561 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:28:37.561 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:28:37.561 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:28:37.561 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:28:37.561 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:28:37.561 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:28:37.601 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:28:38.823 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:30:40.850 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:30:43.460 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:30:45.173 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:30:46.792 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:30:50.075 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:30:50.075 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:30:50.075 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:30:50.075 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:30:50.075 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:30:50.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:30:50.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:30:51.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:30:54.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:30:55.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:30:57.108 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:31:00.305 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:31:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:31:03.919 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:31:03.922 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:31:05.534 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:31:05.534 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:31:06.544 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:31:06.544 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:31:06.544 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:31:06.544 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:31:06.583 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:31:08.064 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:33:09.168 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:33:09.685 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:33:12.246 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:33:13.672 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:33:14.316 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:33:15.209 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:33:15.676 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:33:15.677 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:33:15.677 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:33:16.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:33:17.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:33:18.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:33:19.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:33:23.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:33:26.439 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:33:29.113 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:33:30.182 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:33:31.850 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:33:31.857 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:33:32.812 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:33:32.812 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:33:32.812 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:33:32.812 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:33:32.813 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:33:32.813 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:33:32.872 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:33:34.461 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:35:36.361 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:35:37.881 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:35:39.106 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:35:39.807 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:35:40.498 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:35:40.498 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:35:40.498 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:35:40.498 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:35:40.498 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:35:40.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:35:40.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:35:41.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:35:42.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:35:43.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:35:44.016 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:35:46.113 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:35:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:35:48.487 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:35:48.491 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:35:50.412 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:35:50.412 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:35:50.412 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:35:50.412 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:35:50.413 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:35:50.413 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:35:50.454 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:35:52.061 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:37:53.991 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:37:55.610 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:37:56.888 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:37:59.371 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:38:02.557 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:38:02.560 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:38:02.560 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:38:02.560 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:38:02.560 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:38:02.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:38:02.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:38:04.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:38:06.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:38:07.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:38:09.685 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:38:15.791 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:38:17.399 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:38:20.546 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:38:20.550 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:38:22.037 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:38:22.037 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:38:22.038 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:38:22.039 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:38:22.039 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:38:22.039 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:38:22.084 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:38:22.934 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:40:23.803 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:40:25.302 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:40:26.026 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:40:27.546 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:40:29.168 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:40:29.168 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:40:29.764 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:40:29.765 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:40:29.765 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:40:29.765 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:40:29.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:40:31.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:40:32.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:40:33.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:40:35.678 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:40:40.574 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:40:42.647 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:40:47.419 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:40:47.422 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:40:47.422 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:40:47.422 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:40:47.422 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:40:47.423 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:40:47.423 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:40:49.382 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:40:49.465 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:40:51.480 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:42:52.806 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:42:55.030 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:42:56.021 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:42:58.780 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:43:00.190 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:43:00.191 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:43:00.191 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:43:00.191 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:43:00.191 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:43:00.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:43:00.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:43:02.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:43:04.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:43:05.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:43:08.699 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:43:12.498 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:43:15.186 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:43:16.399 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:43:16.403 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:43:19.109 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:43:19.109 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:43:19.110 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:43:19.110 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:43:19.724 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:43:19.724 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:43:19.763 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:43:21.984 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:45:22.972 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:45:25.149 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:45:27.143 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:45:30.594 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:45:32.164 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:45:32.164 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:45:32.164 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:45:32.164 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:45:32.164 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:45:32.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:45:32.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:45:34.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:45:36.357 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:45:38.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:45:40.488 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:45:42.886 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:45:44.801 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:45:48.681 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:45:48.685 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:45:51.051 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:45:51.051 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:45:51.051 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:45:51.052 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:45:51.052 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:45:54.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:45:54.467 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:45:57.026 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:47:58.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:47:59.355 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:48:00.335 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:48:01.141 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:48:01.993 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:48:01.993 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:48:01.993 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:48:01.993 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:48:01.993 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:48:01.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:48:02.173 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:48:08.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:48:10.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:48:13.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:48:15.318 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:48:17.846 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:48:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:48:20.536 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:48:20.542 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:48:21.541 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:48:21.542 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:48:21.542 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:48:21.542 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:48:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:48:21.542 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:48:21.588 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:48:23.113 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:50:23.609 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:50:23.647 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:50:24.617 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:50:25.597 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:50:26.661 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:50:26.661 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:50:26.661 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:50:26.661 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:50:26.662 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:50:26.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:50:27.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:50:29.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:50:31.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:50:33.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:50:35.387 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:50:38.503 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:50:40.285 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:50:42.673 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:50:42.676 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:50:42.677 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:50:42.677 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:50:42.677 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:50:42.677 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:50:42.678 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:50:42.678 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:50:42.722 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:50:44.247 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:52:46.620 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:52:50.511 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:52:51.894 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:52:53.424 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:52:55.221 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:52:55.221 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:52:55.221 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:52:55.222 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:52:55.222 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:52:55.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:52:55.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:52:57.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:52:59.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:53:01.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:53:05.094 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:53:07.172 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:53:08.646 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:53:10.919 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:53:10.922 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:53:10.923 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:53:10.923 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:53:10.923 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:53:10.923 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:53:10.923 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:53:10.923 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:53:10.961 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:53:12.739 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:55:14.605 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:55:17.122 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:55:18.369 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:55:19.680 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:55:21.557 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:55:21.557 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:55:21.557 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:55:21.557 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:55:21.557 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:55:21.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:55:21.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:55:25.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:55:28.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:55:30.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:55:31.933 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:55:33.691 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:55:35.659 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:55:37.825 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:55:37.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:55:39.238 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:55:39.239 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:55:39.239 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:55:39.239 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:55:39.683 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:55:39.683 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:55:39.724 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:55:40.705 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:57:43.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T18:57:45.099 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:57:48.552 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T18:57:50.611 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T18:57:52.238 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T18:57:52.238 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T18:57:52.238 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T18:57:52.820 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T18:57:52.821 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:57:52.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T18:57:52.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T18:57:54.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T18:57:55.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T18:57:57.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T18:57:58.338 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:58:02.292 INFO:teuthology.misc:Compressing logs...
2023-03-18T18:58:04.587 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:58:06.364 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:58:06.368 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T18:58:06.369 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:58:06.369 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:58:06.369 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:58:07.684 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:58:07.684 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T18:58:07.684 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T18:58:07.731 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T18:58:09.398 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:00:10.759 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:00:12.774 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:00:17.085 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:00:21.041 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:00:24.203 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:00:24.203 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:00:24.203 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:00:24.203 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:00:24.203 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:00:24.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:00:24.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:00:29.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:00:31.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:00:32.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:00:34.502 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:00:36.018 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:00:38.953 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:00:41.181 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:00:41.187 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:00:43.654 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:00:43.654 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:00:43.654 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:00:43.655 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:00:43.655 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:00:43.655 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:00:43.696 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:00:45.295 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:02:45.769 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:02:46.875 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:02:48.787 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:02:52.963 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:02:55.914 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:02:55.914 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:02:55.914 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:02:57.023 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:02:57.023 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:02:57.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:02:57.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:02:58.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:03:01.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:03:02.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:03:06.128 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:03:09.157 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:03:10.886 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:03:11.595 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:03:11.600 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:03:12.484 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:03:12.484 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:03:12.484 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:03:12.485 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:03:12.485 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:03:12.485 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:03:13.008 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:03:13.459 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:05:13.597 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:05:14.125 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:05:15.502 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:05:15.872 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:05:17.155 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:05:17.155 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:05:17.156 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:05:17.156 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:05:17.156 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:05:17.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:05:17.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:05:18.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:05:20.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:05:21.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:05:22.270 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:05:25.132 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:05:26.610 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:05:28.829 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:05:30.059 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:05:31.133 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:05:31.134 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:05:31.134 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:05:31.134 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:05:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:05:31.134 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:05:31.174 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:05:32.229 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:07:33.300 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:07:35.170 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:07:37.860 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:07:38.907 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:07:40.030 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:07:40.030 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:07:40.030 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:07:40.030 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:07:40.031 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:07:40.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:07:41.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:07:43.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:07:45.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:07:50.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:07:51.905 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:07:53.875 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:07:54.998 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:07:56.409 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:07:56.413 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:07:56.414 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:07:56.414 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:07:56.414 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:07:56.414 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:07:56.414 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:07:57.279 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:07:57.323 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:08:00.578 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:10:02.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:10:03.280 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:10:04.408 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:10:06.364 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:10:08.023 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:10:08.023 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:10:09.471 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:10:09.471 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:10:09.472 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:10:09.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:10:11.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:10:13.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:10:16.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:10:23.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:10:26.157 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:10:28.453 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:10:28.483 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:10:30.703 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:10:30.706 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:10:30.706 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:10:30.706 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:10:30.707 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:10:30.707 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:10:33.774 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:10:35.532 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:12:37.020 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:12:40.198 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:12:43.228 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:12:46.527 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:12:48.437 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:12:48.437 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:12:48.438 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:12:48.438 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:12:48.438 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:12:48.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:12:48.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:12:50.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:12:53.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:12:56.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:13:00.139 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:13:02.726 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:13:05.056 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:13:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:13:08.591 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:13:11.175 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:13:12.538 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:13:12.538 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:13:12.538 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:13:12.538 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:13:12.539 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:13:13.083 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:13:14.979 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:15:16.335 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:15:16.869 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:15:18.240 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:15:19.507 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:15:20.689 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:15:20.689 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:15:20.689 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:15:20.689 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:15:20.689 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:15:20.689 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:15:20.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:15:22.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:15:24.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:15:25.705 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:15:27.184 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:15:29.174 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:15:29.917 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:15:31.098 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:15:31.103 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:15:31.607 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:15:32.526 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:15:32.527 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:15:32.527 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:15:33.631 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:15:33.631 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:15:33.672 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:15:35.812 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:17:37.339 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:17:38.458 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:17:43.371 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:17:45.632 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:17:46.846 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:17:46.846 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:17:46.846 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:17:46.846 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:17:46.847 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:17:46.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:17:47.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:17:48.133 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:17:49.002 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:17:50.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:17:51.997 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:17:56.193 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:17:57.682 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:17:58.989 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:17:58.993 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:18:00.607 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:18:00.607 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:18:00.607 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:18:00.608 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:18:00.608 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:18:00.608 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:18:00.646 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:18:03.354 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:20:06.957 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:20:09.427 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:20:11.434 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:20:13.530 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:20:16.206 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:20:16.207 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:20:16.207 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:20:16.207 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:20:16.207 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:20:16.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:20:18.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:20:20.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:20:20.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:20:22.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:20:24.046 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:20:26.182 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:20:26.212 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:20:27.922 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:20:27.926 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:20:30.945 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:20:30.945 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:20:30.945 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:20:30.948 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:20:32.334 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:20:33.540 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:20:35.220 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:20:36.245 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:22:36.313 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:22:37.797 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:22:38.955 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:22:39.928 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:22:41.546 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:22:41.546 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:22:41.546 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:22:41.547 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:22:41.547 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:22:41.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:22:41.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:22:42.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:22:43.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:22:45.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:22:46.390 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:22:47.651 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:22:50.516 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:22:52.393 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:22:52.396 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:22:52.397 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:22:52.397 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:22:52.397 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:22:52.397 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:22:52.397 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:22:52.397 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:22:52.435 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:22:54.424 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:24:57.778 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:25:01.673 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:25:02.904 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:25:05.110 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:25:07.127 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:25:07.127 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:25:07.127 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:25:07.127 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:25:07.127 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:25:07.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:25:07.303 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:25:08.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:25:11.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:25:13.674 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:25:16.355 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:25:20.232 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:25:21.742 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:25:24.094 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:25:24.098 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:25:24.098 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:25:24.099 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:25:24.099 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:25:24.099 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:25:25.431 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:25:25.431 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:25:25.480 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:25:27.279 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:27:28.657 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:27:29.802 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:27:32.381 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:27:35.251 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:27:36.858 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:27:36.858 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:27:36.858 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:27:36.858 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:27:36.858 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:27:37.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:27:37.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:27:38.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:27:39.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:27:40.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:27:41.575 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:27:44.627 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:27:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:27:46.503 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:27:47.615 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:27:48.497 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:27:50.170 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:27:50.171 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:27:50.171 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:27:50.171 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:27:50.171 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:27:50.211 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:27:51.108 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:29:52.066 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:29:53.796 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:29:56.773 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:29:58.508 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:29:59.590 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:29:59.590 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:29:59.590 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:29:59.590 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:29:59.590 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:29:59.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:29:59.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:30:01.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:30:02.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:30:03.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:30:04.893 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:30:06.320 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:30:08.836 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:30:11.143 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:30:11.148 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:30:13.426 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:30:14.895 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:30:14.896 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:30:18.110 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:30:18.111 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:30:18.111 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:30:18.151 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:30:19.728 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:32:22.442 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:32:24.865 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:32:26.880 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:32:28.484 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:32:30.720 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:32:30.720 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:32:30.720 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:32:30.720 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:32:30.720 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:32:30.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:32:30.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:32:35.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:32:37.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:32:39.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:32:41.390 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:32:42.878 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:32:44.768 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:32:45.957 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:32:45.960 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:32:45.960 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:32:45.961 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:32:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:32:45.961 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:32:45.961 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:32:45.961 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:32:45.999 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:32:46.952 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:34:47.961 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:34:49.229 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:34:50.437 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:34:51.366 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:34:52.141 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:34:53.110 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:34:53.110 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:34:53.111 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:34:53.111 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:34:53.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:34:54.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:34:56.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:34:58.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:35:01.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:35:03.699 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:35:05.940 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:35:07.280 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:35:08.617 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:35:08.623 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:35:09.892 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:35:09.892 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:35:09.892 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:35:09.892 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:35:09.893 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:35:09.893 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:35:09.934 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:35:11.729 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:37:13.649 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:37:14.454 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:37:14.631 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:37:15.286 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:37:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:37:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:37:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:37:15.854 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:37:16.480 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:37:16.481 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:37:16.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:37:17.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:37:19.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:37:20.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:37:22.534 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:37:24.032 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:37:24.784 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:37:26.370 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:37:26.374 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:37:26.375 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:37:28.531 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:37:28.531 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:37:28.532 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:37:28.532 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:37:28.532 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:37:28.573 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:37:31.403 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:39:36.451 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:39:38.100 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:39:39.931 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:39:42.185 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:39:44.195 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:39:44.195 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:39:44.195 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:39:44.195 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:39:44.196 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:39:44.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:39:44.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:39:45.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:39:47.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:39:48.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:39:50.660 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:39:53.328 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:39:53.804 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:39:56.075 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:39:56.080 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:39:57.881 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:39:57.881 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:39:57.881 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:39:57.881 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:39:57.882 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:39:57.882 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:39:57.972 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:39:59.756 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:42:02.097 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:42:03.933 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:42:04.511 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:42:05.547 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:42:06.120 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:42:06.121 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:42:06.121 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:42:06.121 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:42:06.122 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:42:06.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:42:06.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:42:06.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:42:07.525 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:42:08.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:42:09.495 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:42:10.812 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:42:12.558 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:42:13.975 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:42:13.981 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:42:14.501 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:42:15.548 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:42:15.548 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:42:15.549 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:42:15.549 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:42:15.549 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:42:15.589 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:42:17.564 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:44:20.474 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:44:21.682 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:44:24.223 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:44:26.543 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:44:27.876 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:44:27.876 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:44:27.876 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:44:27.876 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:44:27.876 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:44:27.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:44:28.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:44:29.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:44:31.327 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:44:34.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:44:37.826 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:44:39.459 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:44:41.843 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:44:42.679 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:44:42.682 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:44:45.420 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:44:45.420 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:44:45.420 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:44:45.420 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:44:45.420 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:44:45.420 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:44:45.468 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:44:46.988 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:46:48.259 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:46:49.527 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:46:50.836 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:46:52.655 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:46:54.134 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:46:55.624 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:46:55.625 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:46:55.625 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:46:55.625 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:46:55.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:46:55.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:46:58.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:47:00.582 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:47:04.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:47:05.919 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:47:07.397 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:47:08.963 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:47:10.365 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:47:10.369 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:47:10.369 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:47:10.369 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:47:10.369 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:47:10.370 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:47:10.370 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:47:10.370 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:47:10.409 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:47:11.404 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:49:12.530 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:49:14.001 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:49:15.708 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:49:17.340 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:49:18.837 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:49:18.837 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:49:18.837 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:49:18.837 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:49:18.838 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:49:18.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:49:19.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:49:19.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:49:21.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:49:24.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:49:25.229 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:49:26.720 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:49:27.917 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:49:28.807 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:49:28.810 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:49:29.481 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:49:29.482 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:49:29.482 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:49:30.334 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:49:30.335 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:49:30.335 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:49:31.590 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:49:35.203 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:51:37.765 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:51:38.035 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:51:38.646 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:51:39.923 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:51:40.589 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:51:40.589 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:51:40.589 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:51:40.589 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:51:40.590 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:51:40.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:51:40.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:51:41.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:51:42.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:51:43.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:51:45.471 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:51:47.110 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:51:48.389 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:51:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:51:49.956 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:51:49.956 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:51:49.956 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:51:49.957 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:51:49.957 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:51:49.957 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:51:51.152 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:51:52.185 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:51:53.261 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:53:54.887 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:53:56.362 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:54:01.754 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:54:05.751 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:54:08.496 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:54:08.496 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:54:08.496 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:54:08.497 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:54:08.497 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:54:09.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:54:11.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:54:13.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:54:15.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:54:18.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:54:22.338 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:54:24.776 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:54:28.076 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:54:29.500 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:54:29.504 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:54:32.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:54:32.322 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:54:35.040 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:54:35.040 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:54:36.109 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:54:36.825 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:54:36.866 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:54:39.936 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:56:43.143 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:56:44.936 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:56:49.647 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:56:52.655 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:56:55.103 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:56:55.104 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:56:55.104 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:56:55.104 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:56:55.104 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:56:55.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:56:55.283 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:56:57.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:57:00.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:57:03.922 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:57:07.341 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:57:10.018 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:57:10.046 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:57:13.500 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:57:13.504 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:57:13.504 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:57:13.504 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:57:13.504 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:57:13.506 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:57:15.591 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:57:18.221 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:59:20.036 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T19:59:22.187 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:59:25.820 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T19:59:27.002 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T19:59:27.848 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T19:59:27.848 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T19:59:27.849 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T19:59:27.849 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T19:59:27.849 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:59:27.849 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T19:59:28.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T19:59:28.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T19:59:31.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T19:59:36.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T19:59:37.747 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:59:45.008 INFO:teuthology.misc:Compressing logs...
2023-03-18T19:59:46.552 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:59:48.219 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:59:48.225 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T19:59:49.837 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:59:49.837 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:59:49.838 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:59:52.927 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:59:52.928 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T19:59:52.928 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T19:59:52.967 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T19:59:53.862 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:01:54.920 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:01:56.462 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:01:57.404 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:01:59.744 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:02:01.527 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:02:01.528 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:02:01.528 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:02:01.528 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:02:01.528 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:02:07.089 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:02:07.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:02:09.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:02:13.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:02:15.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:02:17.903 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:02:21.533 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:02:22.193 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:02:25.253 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:02:25.257 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:02:26.931 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:02:26.932 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:02:26.932 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:02:26.932 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:02:26.932 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:02:26.932 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:02:26.971 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:02:27.745 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:04:29.169 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:04:30.857 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:04:32.195 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:04:33.640 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:04:35.309 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:04:36.939 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:04:37.745 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:04:37.745 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:04:37.745 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:04:39.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:04:39.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:04:41.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:04:42.686 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:04:44.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:04:47.548 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:04:49.896 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:04:52.298 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:04:54.816 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:04:54.819 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:04:55.537 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:04:56.729 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:04:56.730 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:04:56.730 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:04:56.730 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:04:56.730 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:04:56.774 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:04:58.237 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:06:59.293 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:07:00.407 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:07:00.986 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:07:01.406 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:07:01.729 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:07:01.729 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:07:01.729 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:07:01.729 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:07:01.729 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:07:01.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:07:01.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:07:02.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:07:02.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:07:03.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:07:04.167 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:07:05.333 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:07:06.424 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:07:07.500 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:07:07.504 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:07:08.784 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:07:08.785 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:07:08.785 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:07:08.785 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:07:08.785 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:07:08.785 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:07:08.829 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:07:09.836 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:09:11.413 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:09:11.729 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:09:12.203 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:09:13.073 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:09:14.221 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:09:14.221 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:09:14.221 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:09:14.221 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:09:14.222 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:09:14.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:09:14.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:09:15.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:09:18.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:09:19.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:09:20.541 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:09:22.508 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:09:23.660 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:09:25.425 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:09:25.429 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:09:27.385 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:09:27.385 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:09:27.386 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:09:28.966 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:09:28.966 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:09:28.966 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:09:29.057 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:09:30.611 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:11:33.143 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:11:35.326 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:11:38.397 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:11:39.899 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:11:41.275 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:11:42.410 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:11:42.411 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:11:42.411 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:11:42.412 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:11:42.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:11:42.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:11:43.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:11:44.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:11:45.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:11:47.063 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:11:52.110 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:11:52.998 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:11:53.983 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:11:53.986 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:11:55.400 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:11:55.400 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:11:55.400 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:11:55.400 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:11:55.400 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:11:56.653 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:11:56.693 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:11:57.660 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:13:58.820 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:13:58.871 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:13:59.422 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:14:00.373 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:14:01.779 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:14:01.779 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:14:01.779 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:14:01.779 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:14:01.779 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:14:01.780 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:14:01.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:14:03.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:14:07.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:14:09.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:14:11.370 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:14:12.979 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:14:14.001 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:14:15.664 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:14:15.671 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:14:18.555 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:14:18.555 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:14:18.555 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:14:18.556 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:14:18.556 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:14:18.556 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:14:18.595 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:14:21.910 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:16:24.388 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:16:25.501 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:16:26.525 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:16:27.735 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:16:29.060 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:16:29.060 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:16:29.061 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:16:29.061 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:16:29.061 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:16:29.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:16:29.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:16:30.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:16:32.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:16:33.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:16:34.738 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:16:37.189 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:16:38.468 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:16:39.927 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:16:39.930 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:16:41.720 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:16:41.720 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:16:41.720 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:16:41.720 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:16:41.721 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:16:41.721 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:16:41.761 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:16:43.133 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:18:44.990 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:18:46.068 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:18:47.752 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:18:48.120 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:18:49.729 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:18:50.533 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:18:50.533 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:18:51.213 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:18:51.214 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:18:52.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:18:52.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:18:53.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:18:54.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:18:55.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:18:56.503 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:18:58.155 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:18:59.221 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:19:00.604 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:19:00.608 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:19:00.608 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:19:00.609 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:19:00.609 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:19:00.609 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:19:00.609 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:19:00.609 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:19:00.647 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:19:01.873 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:21:05.001 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:21:06.244 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:21:09.667 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:21:11.667 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:21:13.346 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:21:13.346 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:21:13.346 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:21:13.346 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:21:13.347 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:21:13.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:21:13.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:21:15.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:21:18.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:21:20.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:21:22.901 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:21:24.785 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:21:25.629 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:21:26.797 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:21:26.802 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:21:28.435 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:21:29.255 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:21:29.255 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:21:29.255 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:21:29.255 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:21:29.255 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:21:29.295 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:21:30.695 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:23:32.254 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:23:35.602 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:23:36.740 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:23:38.070 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:23:40.338 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:23:40.338 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:23:40.338 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:23:40.338 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:23:40.338 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:23:40.338 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:23:40.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:23:42.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:23:45.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:23:50.733 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:23:54.932 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:23:57.561 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:23:59.866 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:24:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:24:01.318 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:24:03.260 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:24:03.261 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:24:03.261 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:24:04.241 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:24:04.241 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:24:04.689 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:24:04.749 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:24:05.471 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:26:07.155 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:26:07.622 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:26:08.889 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:26:09.907 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:26:10.870 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:26:10.870 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:26:10.870 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:26:10.870 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:26:10.870 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:26:10.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:26:11.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:26:11.231 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:26:12.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:26:14.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:26:17.702 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:26:20.072 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:26:23.019 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:26:25.682 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:26:25.685 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:26:25.686 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:26:25.686 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:26:25.686 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:26:25.686 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:26:27.554 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:26:27.554 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:26:27.598 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:26:29.209 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:28:30.285 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:28:31.033 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:28:32.011 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:28:33.040 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:28:33.857 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:28:33.857 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:28:33.857 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:28:33.857 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:28:33.857 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:28:35.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:28:35.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:28:36.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:28:37.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:28:38.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:28:41.045 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:28:42.011 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:28:43.426 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:28:44.938 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:28:44.943 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:28:46.432 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:28:46.432 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:28:46.432 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:28:46.432 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:28:46.432 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:28:46.432 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:28:46.471 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:28:47.580 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:30:49.044 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:30:50.704 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:30:52.941 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:30:54.345 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:30:56.160 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:30:56.160 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:30:56.161 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:30:56.161 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:30:56.161 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:30:56.161 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:30:58.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:31:01.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:31:03.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:31:05.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:31:07.054 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:31:08.616 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:31:09.855 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:31:12.913 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:31:12.918 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:31:12.919 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:31:12.919 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:31:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:31:14.601 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:31:14.601 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:31:14.601 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:31:14.640 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:31:17.468 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:33:18.720 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:33:22.362 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:33:24.603 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:33:27.366 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:33:29.000 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:33:29.000 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:33:29.000 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:33:29.000 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:33:29.001 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:33:29.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:33:29.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:33:31.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:33:33.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:33:35.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:33:37.561 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:33:39.302 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:33:41.785 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:33:42.865 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:33:43.913 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:33:43.914 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:33:43.914 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:33:43.914 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:33:44.823 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:33:44.823 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:33:44.823 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:33:44.862 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:33:45.936 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:35:47.072 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:35:48.544 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:35:49.657 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:35:51.640 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:35:53.906 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:35:53.906 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:35:53.906 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:35:53.907 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:35:53.907 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:35:53.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:35:54.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:35:55.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:35:57.974 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:36:00.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:36:04.838 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:36:06.370 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:36:07.567 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:36:09.214 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:36:09.218 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:36:10.504 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:36:10.504 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:36:10.505 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:36:10.505 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:36:10.505 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:36:10.505 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:36:10.544 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:36:12.449 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:38:13.326 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-18T20:38:16.521 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:38:17.908 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:38:18.559 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-18T20:38:20.752 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-18T20:38:20.753 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-18T20:38:20.753 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-18T20:38:20.753 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-18T20:38:20.753 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:38:20.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com'
2023-03-18T20:38:20.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com'
2023-03-18T20:38:22.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2023-03-18T20:38:23.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com'
2023-03-18T20:38:25.920 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com'
2023-03-18T20:38:28.292 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:38:30.838 INFO:teuthology.misc:Compressing logs...
2023-03-18T20:38:32.721 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:38:34.181 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:38:34.186 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi102 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-18T20:38:37.231 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:38:37.231 INFO:teuthology.orchestra.run.smithi181.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:38:37.231 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:38:37.231 INFO:teuthology.orchestra.run.smithi159.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:38:37.231 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists;	not overwritten
2023-03-18T20:38:37.232 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /var/log/ceph/bdd74dd2-c540-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists;	not overwritten
2023-03-18T20:38:37.317 INFO:teuthology.kill:Killing Pids: {13923}
2023-03-18T20:38:39.602 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_created', 'teuthology_job_results_total', 'teuthology_job_results'}
2023-03-18T20:40:40.776 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-18T20:40:42.600 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-18T20:40:43.735 INFO:teuthology.nuke:Checking targets against current locks
2023-03-18T20:40:43.796 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T20:40:46.379 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T20:40:47.894 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T20:40:49.890 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T20:40:51.927 INFO:teuthology.orchestra.console:Power off smithi086
2023-03-18T20:40:53.143 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-18T20:40:55.488 INFO:teuthology.orchestra.console:Power off smithi102
2023-03-18T20:40:58.258 INFO:teuthology.orchestra.console:Power off smithi141
2023-03-18T20:41:01.172 INFO:teuthology.orchestra.console:Power off smithi159
2023-03-18T20:41:04.063 INFO:teuthology.orchestra.console:Power off smithi181
2023-03-18T20:41:06.371 INFO:teuthology.orchestra.console:Power off for smithi086 completed
2023-03-18T20:41:07.896 INFO:teuthology.orchestra.console:Power off for smithi102 completed
2023-03-18T20:41:08.064 INFO:teuthology.lock.ops:unlocked: smithi102.front.sepia.ceph.com
2023-03-18T20:41:09.923 INFO:teuthology.lock.ops:unlocked: smithi086.front.sepia.ceph.com
2023-03-18T20:41:15.339 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2023-03-18T20:41:16.147 INFO:teuthology.orchestra.console:Power off for smithi159 completed
2023-03-18T20:41:16.251 INFO:teuthology.orchestra.console:Power off for smithi181 completed
2023-03-18T20:41:16.763 INFO:teuthology.lock.ops:unlocked: smithi141.front.sepia.ceph.com
2023-03-18T20:41:18.165 INFO:teuthology.lock.ops:unlocked: smithi159.front.sepia.ceph.com
2023-03-18T20:41:18.828 INFO:teuthology.lock.ops:unlocked: smithi181.front.sepia.ceph.com