2023-03-18T13:58:15.264 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f
2023-03-18T13:58:17.399 INFO:teuthology.lock.ops:Start node 'smithi027.front.sepia.ceph.com' reimaging
2023-03-18T13:58:19.175 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T13:58:20.481 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2023-03-18T13:58:24.337 INFO:teuthology.lock.ops:Node 'smithi027.front.sepia.ceph.com' reimaging is complete
2023-03-18T13:58:25.550 INFO:teuthology.lock.ops:Start node 'smithi037.front.sepia.ceph.com' reimaging
2023-03-18T13:58:25.550 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T13:58:25.550 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2023-03-18T13:58:25.578 INFO:teuthology.lock.ops:Node 'smithi037.front.sepia.ceph.com' reimaging is complete
2023-03-18T13:58:27.055 INFO:teuthology.lock.ops:Start node 'smithi103.front.sepia.ceph.com' reimaging
2023-03-18T13:58:27.056 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T13:58:27.056 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-03-18T13:58:27.087 INFO:teuthology.lock.ops:Node 'smithi103.front.sepia.ceph.com' reimaging is complete
2023-03-18T13:58:28.978 INFO:teuthology.lock.ops:Start node 'smithi187.front.sepia.ceph.com' reimaging
2023-03-18T13:58:28.978 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T13:58:28.978 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2023-03-18T13:58:29.002 INFO:teuthology.lock.ops:Node 'smithi187.front.sepia.ceph.com' reimaging is complete
2023-03-18T13:58:31.620 INFO:teuthology.lock.ops:Start node 'smithi188.front.sepia.ceph.com' reimaging
2023-03-18T13:58:31.620 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: reset os type and version on server
2023-03-18T13:58:34.928 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server
2023-03-18T13:58:34.951 INFO:teuthology.lock.ops:Node 'smithi188.front.sepia.ceph.com' reimaging is complete
2023-03-18T13:58:36.398 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of centos 8.stream
2023-03-18T13:58:39.968 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of centos 8.stream
2023-03-18T13:58:41.913 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of centos 8.stream
2023-03-18T13:58:41.987 INFO:teuthology.provision.fog.smithi187:Scheduling deploy of centos 8.stream
2023-03-18T13:58:44.683 INFO:teuthology.provision.fog.smithi188:Scheduling deploy of centos 8.stream
2023-03-18T13:58:47.181 INFO:teuthology.orchestra.console:Power off smithi027
2023-03-18T13:58:49.998 INFO:teuthology.orchestra.console:Power off smithi103
2023-03-18T13:58:52.456 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2023-03-18T13:58:54.482 INFO:teuthology.orchestra.console:Power off smithi037
2023-03-18T13:58:55.775 INFO:teuthology.orchestra.console:Power on smithi027
2023-03-18T13:58:57.477 INFO:teuthology.orchestra.console:Power off smithi187
2023-03-18T13:58:58.916 INFO:teuthology.orchestra.console:Power off smithi188
2023-03-18T13:59:00.473 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2023-03-18T13:59:02.373 INFO:teuthology.orchestra.console:Power on smithi037
2023-03-18T13:59:04.920 INFO:teuthology.orchestra.console:Power off for smithi103 completed
2023-03-18T13:59:06.681 INFO:teuthology.orchestra.console:Power on for smithi027 completed
2023-03-18T13:59:06.782 INFO:teuthology.orchestra.console:Power on smithi103
2023-03-18T13:59:08.113 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish
2023-03-18T13:59:10.190 INFO:teuthology.orchestra.console:Power on for smithi037 completed
2023-03-18T13:59:11.154 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2023-03-18T13:59:11.254 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish
2023-03-18T13:59:12.426 INFO:teuthology.orchestra.console:Power on smithi187
2023-03-18T13:59:13.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T13:59:14.828 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T13:59:17.528 INFO:teuthology.orchestra.console:Power off for smithi188 completed
2023-03-18T13:59:20.155 INFO:teuthology.orchestra.console:Power on for smithi103 completed
2023-03-18T13:59:20.155 INFO:teuthology.orchestra.console:Power on for smithi187 completed
2023-03-18T13:59:20.255 INFO:teuthology.orchestra.console:Power on smithi188
2023-03-18T13:59:21.767 INFO:teuthology.provision.fog.smithi103:Waiting for deploy to finish
2023-03-18T13:59:23.744 INFO:teuthology.provision.fog.smithi187:Waiting for deploy to finish
2023-03-18T13:59:28.008 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T13:59:29.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T13:59:29.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T13:59:29.565 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T13:59:29.826 INFO:teuthology.orchestra.console:Power on for smithi188 completed
2023-03-18T13:59:31.737 INFO:teuthology.provision.fog.smithi188:Waiting for deploy to finish
2023-03-18T13:59:38.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T13:59:39.969 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T13:59:39.969 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T13:59:39.970 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T13:59:47.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T13:59:49.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T13:59:50.672 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T13:59:50.672 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T13:59:58.085 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T13:59:59.749 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:00:01.076 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:00:01.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T14:00:10.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:00:12.373 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:00:12.373 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T14:00:21.445 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:00:24.467 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:00:24.468 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T14:00:24.468 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T14:00:30.595 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:32.998 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:34.911 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:00:34.911 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:00:36.030 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:37.819 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:39.819 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:42.061 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:43.960 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:44.191 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:00:45.267 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:46.782 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:49.109 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T14:00:50.186 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:51.913 WARNING:teuthology.provision.fog:'connect to smithi037.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds
2023-03-18T14:00:54.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:00:56.830 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:58.227 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:00:59.790 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:00:59.858 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:01.228 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:02.358 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:03.371 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:04.403 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:05.175 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T14:01:05.658 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:01:08.004 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T14:01:08.005 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T14:01:08.005 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T14:01:08.005 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T14:01:08.005 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T14:01:08.005 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T14:01:08.005 ERROR:paramiko.transport:EOFError
2023-03-18T14:01:08.005 ERROR:paramiko.transport:
2023-03-18T14:01:08.005 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T14:01:08.005 ERROR:paramiko.transport:
2023-03-18T14:01:08.005 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:01:08.005 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T14:01:08.005 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T14:01:08.005 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T14:01:08.005 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T14:01:08.005 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T14:01:08.005 ERROR:paramiko.transport:
2023-03-18T14:01:08.619 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T14:01:08.687 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:10.482 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:13.757 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:16.069 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:19.579 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:20.846 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:22.684 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:01:25.215 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T14:01:27.146 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:01:27.146 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T14:01:27.146 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T14:01:27.146 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T14:01:27.146 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T14:01:27.146 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T14:01:27.146 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T14:01:27.146 ERROR:paramiko.transport:EOFError
2023-03-18T14:01:27.146 ERROR:paramiko.transport:
2023-03-18T14:01:27.146 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T14:01:27.146 ERROR:paramiko.transport:
2023-03-18T14:01:27.147 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:01:27.147 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T14:01:27.147 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T14:01:27.147 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T14:01:27.147 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T14:01:27.147 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T14:01:27.147 ERROR:paramiko.transport:
2023-03-18T14:01:27.147 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T14:01:27.163 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:28.758 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:29.850 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:30.789 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:32.599 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:34.560 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:37.311 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:38.675 WARNING:teuthology.provision.fog:'connect to smithi187.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds
2023-03-18T14:01:38.737 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:39.996 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:41.080 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:42.075 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:44.032 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:44.893 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:45.802 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:47.389 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:48.782 ERROR:teuthology.orchestra.connection:Error authenticating with smithi187.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:50.212 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:51.839 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T14:01:54.084 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:01:54.084 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T14:01:54.084 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T14:01:54.084 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T14:01:54.084 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T14:01:54.085 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T14:01:54.085 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T14:01:54.085 ERROR:paramiko.transport:EOFError
2023-03-18T14:01:54.085 ERROR:paramiko.transport:
2023-03-18T14:01:54.085 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T14:01:54.085 ERROR:paramiko.transport:
2023-03-18T14:01:54.085 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:01:54.085 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T14:01:54.086 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T14:01:54.086 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T14:01:54.086 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T14:01:54.086 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T14:01:54.086 ERROR:paramiko.transport:
2023-03-18T14:01:54.086 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T14:01:54.099 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:55.389 ERROR:teuthology.orchestra.connection:Error authenticating with smithi103.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:56.683 WARNING:teuthology.provision.fog:'connect to smithi103.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds
2023-03-18T14:01:56.798 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:01:59.343 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:02:02.631 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:02:04.466 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T14:02:04.466 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:02:05.759 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T14:02:05.759 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T14:02:05.759 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T14:02:05.759 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T14:02:05.759 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T14:02:05.759 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T14:02:05.759 ERROR:paramiko.transport:EOFError
2023-03-18T14:02:05.759 ERROR:paramiko.transport:
2023-03-18T14:02:05.759 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T14:02:05.759 ERROR:paramiko.transport:
2023-03-18T14:02:05.760 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:02:05.760 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T14:02:05.760 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T14:02:05.760 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T14:02:05.760 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T14:02:05.760 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T14:02:05.760 ERROR:paramiko.transport:
2023-03-18T14:02:05.760 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T14:02:05.834 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:02:06.353 WARNING:teuthology.provision.fog:'connect to smithi027.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds
2023-03-18T14:02:13.311 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:02:15.926 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:02:20.708 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-18T14:02:21.709 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T14:02:22.965 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:02:22.965 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T14:02:22.965 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T14:02:22.965 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T14:02:22.965 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T14:02:22.965 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T14:02:22.965 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T14:02:22.966 ERROR:paramiko.transport:EOFError
2023-03-18T14:02:22.966 ERROR:paramiko.transport:
2023-03-18T14:02:22.966 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T14:02:22.966 ERROR:paramiko.transport:
2023-03-18T14:02:22.966 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:02:22.966 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T14:02:22.966 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T14:02:22.966 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T14:02:22.966 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T14:02:22.966 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T14:02:24.817 ERROR:paramiko.transport:
2023-03-18T14:02:24.818 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T14:02:29.765 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-18T14:02:31.101 ERROR:teuthology.orchestra.connection:Error authenticating with smithi027.front.sepia.ceph.com: Authentication failed.
2023-03-18T14:02:33.464 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner
2023-03-18T14:02:34.581 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:02:34.581 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner
2023-03-18T14:02:34.581 ERROR:paramiko.transport:    buf = self.packetizer.readline(timeout)
2023-03-18T14:02:34.581 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline
2023-03-18T14:02:36.349 ERROR:paramiko.transport:    buf += self._read_timeout(timeout)
2023-03-18T14:02:36.349 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout
2023-03-18T14:02:36.349 ERROR:paramiko.transport:    raise EOFError()
2023-03-18T14:02:36.349 ERROR:paramiko.transport:EOFError
2023-03-18T14:02:36.349 ERROR:paramiko.transport:
2023-03-18T14:02:36.349 ERROR:paramiko.transport:During handling of the above exception, another exception occurred:
2023-03-18T14:02:36.349 ERROR:paramiko.transport:
2023-03-18T14:02:36.349 ERROR:paramiko.transport:Traceback (most recent call last):
2023-03-18T14:02:36.349 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run
2023-03-18T14:02:36.349 ERROR:paramiko.transport:    self._check_banner()
2023-03-18T14:02:36.350 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner
2023-03-18T14:02:36.350 ERROR:paramiko.transport:    raise SSHException(
2023-03-18T14:02:36.350 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner
2023-03-18T14:02:36.350 ERROR:paramiko.transport:
2023-03-18T14:02:36.350 WARNING:teuthology.provision.fog:Error reading SSH protocol banner
2023-03-18T14:02:37.357 ERROR:teuthology.orchestra.connection:Error authenticating with smithi037.front.sepia.ceph.com: Authentication timeout.
2023-03-18T14:02:39.653 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-18T14:02:44.197 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T14:02:50.089 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-18T14:02:56.101 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.37
2023-03-18T14:03:01.509 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-18T14:03:05.013 WARNING:teuthology.provision.fog:timed out
2023-03-18T14:03:05.455 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T14:03:11.763 WARNING:teuthology.provision.fog:timed out
2023-03-18T14:03:20.965 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:03:22.261 INFO:teuthology.provision.fog.smithi037:Node is ready
2023-03-18T14:03:23.587 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com
2023-03-18T14:03:24.145 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037
2023-03-18T14:03:25.798 INFO:teuthology.provision.fog.smithi037:Deploy complete!
2023-03-18T14:03:27.621 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104)
2023-03-18T14:03:31.292 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-18T14:03:32.911 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.187
2023-03-18T14:03:40.974 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T14:03:43.654 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.188
2023-03-18T14:03:47.246 WARNING:teuthology.provision.fog:timed out
2023-03-18T14:03:49.637 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:03:53.530 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T14:03:56.325 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:04:00.549 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.103
2023-03-18T14:04:02.810 INFO:teuthology.provision.fog.smithi187:Node is ready
2023-03-18T14:04:04.958 INFO:teuthology.orchestra.run.smithi187.stdout:smithi187.front.sepia.ceph.com
2023-03-18T14:04:07.243 INFO:teuthology.provision.fog.smithi188:Node is ready
2023-03-18T14:04:07.259 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.15.187 smithi187.front.sepia.ceph.com smithi187
2023-03-18T14:04:09.225 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.27
2023-03-18T14:04:09.228 INFO:teuthology.orchestra.run.smithi188.stdout:smithi188.front.sepia.ceph.com
2023-03-18T14:04:10.887 INFO:teuthology.orchestra.run.smithi188.stdout:172.21.15.188 smithi188.front.sepia.ceph.com smithi188
2023-03-18T14:04:12.749 INFO:teuthology.provision.fog.smithi187:Deploy complete!
2023-03-18T14:04:13.595 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T14:04:16.469 INFO:teuthology.provision.fog.smithi188:Deploy complete!
2023-03-18T14:04:26.437 INFO:teuthology.provision.fog.smithi103:Node is ready
2023-03-18T14:04:26.818 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103.front.sepia.ceph.com
2023-03-18T14:04:28.818 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103 smithi103.front.sepia.ceph.com smithi103
2023-03-18T14:04:30.433 INFO:teuthology.provision.fog.smithi103:Deploy complete!
2023-03-18T14:05:16.400 WARNING:teuthology.provision.fog:timed out
2023-03-18T14:05:24.286 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-03-18T14:05:24.772 INFO:teuthology.provision.fog.smithi027:Node is ready
2023-03-18T14:05:25.614 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027.front.sepia.ceph.com
2023-03-18T14:05:26.840 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27 smithi027.front.sepia.ceph.com smithi027
2023-03-18T14:05:29.091 INFO:teuthology.provision.fog.smithi027:Deploy complete!
2023-03-18T14:05:31.456 INFO:teuthology.lock.ops:Checking smithi027.front.sepia.ceph.com
2023-03-18T14:05:32.584 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T14:05:32.614 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com
2023-03-18T14:05:34.684 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T14:05:34.922 INFO:teuthology.lock.ops:Checking smithi103.front.sepia.ceph.com
2023-03-18T14:05:37.333 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T14:05:37.363 INFO:teuthology.lock.ops:Checking smithi187.front.sepia.ceph.com
2023-03-18T14:05:41.008 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T14:05:41.037 INFO:teuthology.lock.ops:Checking smithi188.front.sepia.ceph.com
2023-03-18T14:05:43.674 INFO:teuthology.lock.ops:New key found. Updating...
2023-03-18T14:05:43.705 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: set os type and version on server
2023-03-18T14:05:46.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-18T14:05:47.161 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64
2023-03-18T14:05:50.121 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="CentOS Stream"
2023-03-18T14:05:51.627 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8"
2023-03-18T14:05:51.627 INFO:teuthology.orchestra.run.smithi027.stdout:ID="centos"
2023-03-18T14:05:51.627 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="rhel fedora"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://centos.org/"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T14:05:51.628 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-18T14:05:51.631 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server
2023-03-18T14:05:51.661 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server
2023-03-18T14:05:54.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-18T14:05:54.832 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64
2023-03-18T14:05:56.317 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream"
2023-03-18T14:05:57.986 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8"
2023-03-18T14:06:00.035 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos"
2023-03-18T14:06:00.035 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora"
2023-03-18T14:06:00.035 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8"
2023-03-18T14:06:00.035 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8"
2023-03-18T14:06:00.035 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-18T14:06:00.035 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31"
2023-03-18T14:06:00.036 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-18T14:06:00.036 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/"
2023-03-18T14:06:01.937 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T14:06:01.937 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T14:06:01.937 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-18T14:06:01.937 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2023-03-18T14:06:02.409 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: set os type and version on server
2023-03-18T14:06:03.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-18T14:06:03.450 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64
2023-03-18T14:06:03.942 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream"
2023-03-18T14:06:04.673 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="8"
2023-03-18T14:06:04.673 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos"
2023-03-18T14:06:04.673 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora"
2023-03-18T14:06:04.673 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="8"
2023-03-18T14:06:04.673 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el8"
2023-03-18T14:06:04.673 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-18T14:06:04.673 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31"
2023-03-18T14:06:05.632 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-18T14:06:07.430 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/"
2023-03-18T14:06:07.430 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T14:06:07.430 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T14:06:07.430 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-18T14:06:07.431 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server
2023-03-18T14:06:07.459 INFO:teuthology.lock.ops:Updating [smithi187.front.sepia.ceph.com]: set os type and version on server
2023-03-18T14:06:09.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-18T14:06:09.322 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64
2023-03-18T14:06:10.085 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream"
2023-03-18T14:06:11.759 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8"
2023-03-18T14:06:11.759 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos"
2023-03-18T14:06:12.622 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora"
2023-03-18T14:06:12.622 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T14:06:12.958 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-18T14:06:14.861 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server
2023-03-18T14:06:14.891 INFO:teuthology.lock.ops:Updating [smithi188.front.sepia.ceph.com]: set os type and version on server
2023-03-18T14:06:17.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-18T14:06:17.546 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64
2023-03-18T14:06:21.121 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="CentOS Stream"
2023-03-18T14:06:23.226 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="8"
2023-03-18T14:06:23.226 INFO:teuthology.orchestra.run.smithi188.stdout:ID="centos"
2023-03-18T14:06:23.226 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="rhel fedora"
2023-03-18T14:06:23.226 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="8"
2023-03-18T14:06:23.226 INFO:teuthology.orchestra.run.smithi188.stdout:PLATFORM_ID="platform:el8"
2023-03-18T14:06:23.226 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="CentOS Stream 8"
2023-03-18T14:06:24.737 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31"
2023-03-18T14:06:24.737 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-03-18T14:06:24.737 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://centos.org/"
2023-03-18T14:06:24.737 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-03-18T14:06:24.737 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-03-18T14:06:24.737 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-03-18T14:06:24.738 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server
2023-03-18T14:06:24.864 INFO:teuthology.dispatcher.supervisor:Running job 7209621
2023-03-18T14:06:26.269 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209621 --name yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi --description rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools agent/off mon_election/classic task/test_orch_cli_mon} -- /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209621/orig.config.yaml
2023-03-18T14:06:26.276 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-15_23:29:37-rados-wip-yuri2-testing-2023-03-15-1204-quincy-distro-default-smithi/7209621
2023-03-18T14:06:27.292 INFO:teuthology.dispatcher.supervisor:Job PID: 26761
2023-03-18T14:06:31.101 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-03-19T02:06:54.454 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:06:56.927 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:06:58.195 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
KeyError: 'status'
2023-03-19T02:06:59.646 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:07:01.518 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:07:01.518 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:07:01.518 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:07:01.518 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:07:01.518 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:07:03.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:07:03.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:07:05.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:07:08.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:07:10.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:07:13.074 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:07:15.031 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-19T02:07:16.322 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-19T02:07:18.398 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-03-19T02:07:19.126 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:07:26.970 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2023-03-19T02:07:28.246 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.14.log: file size changed while zipping
2023-03-19T02:07:29.554 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-mon.smithi187.log: file size changed while zipping
2023-03-19T02:07:34.872 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.8.log: file size changed while zipping
2023-03-19T02:07:36.134 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.12.log: file size changed while zipping
2023-03-19T02:07:38.671 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.9.log: file size changed while zipping
2023-03-19T02:07:41.558 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-mon.smithi188.log: file size changed while zipping
2023-03-19T02:07:44.555 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.11.log: file size changed while zipping
2023-03-19T02:07:46.760 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2023-03-19T02:07:50.364 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.13.log: file size changed while zipping
2023-03-19T02:07:51.925 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /var/log/ceph/53ce0764-c59a-11ed-9afc-001a4aab830c/ceph-osd.10.log: file size changed while zipping
2023-03-19T02:08:11.573 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:08:14.024 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:10:18.886 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:10:20.220 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:10:21.173 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:10:22.404 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:10:23.720 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:10:23.720 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:10:23.720 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:10:23.720 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:10:23.721 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:10:23.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:10:23.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:10:25.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:10:26.897 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:10:29.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:10:36.531 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:10:38.592 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:10:59.746 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:11:03.986 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:13:05.833 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:13:07.962 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:13:09.212 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:13:10.979 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:13:11.883 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:13:12.972 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:13:12.973 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:13:12.973 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:13:12.973 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:13:12.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:13:13.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:13:13.808 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:13:14.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:13:16.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:13:17.987 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:13:19.913 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:13:41.955 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:13:43.179 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:15:44.808 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:15:47.901 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:15:50.460 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:15:52.189 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:15:55.087 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:15:55.087 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:15:55.088 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:15:55.088 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:15:55.088 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:15:55.088 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:15:55.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:15:56.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:15:58.132 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:16:00.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:16:02.779 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:16:04.639 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:16:26.961 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:16:28.550 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:18:32.214 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:18:34.321 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:18:36.486 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:18:38.538 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:18:40.330 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:18:40.330 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:18:40.330 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:18:40.330 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:18:40.330 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:18:40.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:18:40.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:18:43.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:18:45.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:18:47.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:18:49.846 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:18:52.377 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:19:13.291 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:19:14.543 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:21:16.206 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:21:17.547 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:21:18.774 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:21:20.390 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:21:21.381 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:21:22.821 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:21:22.821 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:21:22.821 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:21:22.821 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:21:22.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:21:22.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:21:24.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:21:26.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:21:29.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:21:32.075 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:21:34.075 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:21:55.055 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:21:55.881 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:23:57.091 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:23:58.866 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:24:00.654 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:24:01.917 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:24:03.434 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:24:04.491 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:24:04.491 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:24:04.491 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:24:04.491 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:24:04.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:24:04.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:24:05.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:24:07.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:24:08.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:24:09.614 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:24:11.102 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:24:32.342 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:24:33.877 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:26:34.873 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:26:36.449 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:26:37.009 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:26:37.650 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:26:37.986 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:26:38.724 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:26:38.724 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:26:38.724 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:26:38.725 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:26:38.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:26:38.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:26:39.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:26:39.985 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:26:40.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:26:41.442 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:26:42.840 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:27:03.736 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:27:05.691 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:29:07.980 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:29:10.189 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:29:11.506 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:29:12.578 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:29:13.309 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:29:13.310 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:29:13.310 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:29:13.310 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:29:13.310 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:29:13.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:29:13.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:29:14.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:29:15.394 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:29:16.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:29:17.461 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:29:19.012 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:29:39.167 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:29:40.368 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:31:43.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:31:44.813 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:31:46.623 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:31:48.537 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:31:50.878 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:31:50.878 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:31:50.878 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:31:50.878 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:31:50.878 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:31:50.878 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:31:51.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:31:52.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:31:55.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:31:57.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:31:58.767 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:32:00.964 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:32:22.997 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:32:24.154 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:34:27.147 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:34:29.193 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:34:31.506 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:34:31.858 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:34:33.013 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:34:33.013 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:34:33.013 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:34:33.013 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:34:33.013 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:34:33.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:34:33.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:34:34.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:34:35.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:34:37.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:34:39.036 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:34:40.523 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:35:01.037 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:35:02.529 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:37:04.257 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:37:06.892 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:37:09.007 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:37:09.873 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:37:10.699 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:37:10.699 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:37:10.699 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:37:10.699 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:37:10.699 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:37:10.699 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:37:10.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:37:11.602 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:37:11.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:37:12.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:37:14.722 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:37:19.630 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:37:40.443 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:37:42.108 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:39:44.181 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:39:45.932 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:39:48.472 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:39:49.895 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:39:51.215 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:39:51.216 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:39:51.216 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:39:51.216 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:39:51.216 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:39:51.216 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:39:51.395 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:39:53.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:39:55.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:39:57.024 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:39:58.581 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:40:01.555 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:40:25.193 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:40:26.397 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:42:28.058 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:42:29.545 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:42:30.866 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:42:31.262 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:42:32.418 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:42:32.418 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:42:32.418 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:42:32.418 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:42:33.139 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:42:33.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:42:33.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:42:33.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:42:34.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:42:35.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:42:37.183 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:42:39.080 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:43:00.391 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:43:01.971 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:45:04.550 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:45:05.489 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:45:06.507 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:45:08.431 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:45:10.074 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:45:10.074 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:45:10.074 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:45:10.074 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:45:10.074 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:45:10.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:45:10.258 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:45:13.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:45:14.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:45:16.016 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:45:16.919 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:45:19.917 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:45:42.171 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:45:42.526 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:47:46.610 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:47:48.646 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:47:50.627 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:47:52.325 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:47:53.777 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:47:53.778 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:47:53.778 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:47:53.778 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:47:53.778 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:47:55.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:47:55.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:47:58.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:47:59.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:48:02.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:48:04.533 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:48:06.530 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:48:27.049 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:48:28.069 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:50:29.734 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:50:31.829 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:50:33.982 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:50:36.542 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:50:38.328 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:50:38.328 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:50:38.328 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:50:38.328 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:50:38.329 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:50:38.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:50:38.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:50:39.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:50:41.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:50:44.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:50:46.831 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:50:49.551 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:51:10.454 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:51:13.806 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:53:15.250 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:53:16.327 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:53:18.031 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:53:20.328 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:53:21.356 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:53:21.356 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:53:21.356 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:53:21.356 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:53:21.356 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:53:21.356 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:53:21.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:53:23.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:53:27.236 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:53:30.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:53:31.375 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:53:32.849 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:53:53.888 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:53:55.679 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:55:58.224 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:56:02.193 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:56:06.662 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:56:08.643 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:56:10.053 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:56:10.054 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:56:10.054 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:56:10.054 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:56:10.054 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:56:10.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:56:10.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:56:11.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:56:13.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:56:15.229 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:56:16.590 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:56:18.142 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:56:38.755 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:56:42.191 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:58:45.140 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T02:58:46.973 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:58:50.838 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T02:58:53.540 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T02:58:55.123 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T02:58:58.293 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T02:58:58.293 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T02:58:58.293 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T02:58:58.293 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:58:58.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T02:58:58.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T02:59:01.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T02:59:02.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T02:59:06.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T02:59:07.828 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:59:09.528 INFO:teuthology.misc:Compressing logs...
2023-03-19T02:59:31.197 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T02:59:40.794 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:01:42.504 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:01:45.599 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:01:47.444 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:01:49.094 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:01:50.612 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:01:50.612 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:01:51.022 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:01:51.022 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:01:51.022 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:01:52.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:01:52.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:01:53.593 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:01:54.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:01:55.376 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:01:57.382 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:02:00.272 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:02:21.163 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:02:21.914 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:04:24.287 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:04:25.650 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:04:26.296 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:04:27.646 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:04:29.158 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:04:29.158 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:04:29.158 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:04:31.014 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:04:31.014 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:04:33.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:04:33.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:04:34.154 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:04:36.090 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:04:37.503 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:04:39.106 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:04:40.679 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:05:01.492 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:05:03.031 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:07:05.646 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:07:06.942 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:07:08.018 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:07:10.000 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:07:10.878 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:07:10.879 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:07:12.066 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:07:13.953 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:07:13.953 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:07:13.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:07:14.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:07:15.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:07:17.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:07:19.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:07:23.700 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:07:27.337 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:07:51.051 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:07:53.390 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:09:57.384 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:09:59.157 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:10:00.484 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:10:01.512 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:10:03.690 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:10:03.690 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:10:03.690 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:10:03.690 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:10:03.690 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:10:03.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:10:05.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:10:07.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:10:09.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:10:10.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:10:14.060 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:10:16.514 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:10:37.258 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:10:39.027 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:12:43.333 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:12:45.017 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:12:46.835 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:12:49.412 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:12:50.445 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:12:50.445 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:12:50.445 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:12:50.445 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:12:50.445 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:12:50.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:12:50.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:12:54.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:12:56.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:12:58.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:13:00.157 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:13:01.932 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:13:23.761 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:13:26.970 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:15:28.063 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:15:29.772 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:15:34.171 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:15:37.836 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:15:40.358 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:15:40.358 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:15:43.150 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:15:43.150 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:15:43.150 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:15:43.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:15:43.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:15:44.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:15:45.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:15:47.297 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:15:48.436 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:15:49.896 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:16:10.465 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:16:11.684 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:18:13.838 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:18:14.764 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:18:17.702 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:18:20.749 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:18:23.310 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:18:23.310 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:18:23.311 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:18:23.311 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:18:23.311 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:18:23.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:18:23.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:18:27.410 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:18:29.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:18:31.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:18:34.002 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:18:35.217 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:18:37.012 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:18:38.620 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:18:41.756 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:20:43.215 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:20:44.937 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:20:47.103 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:20:49.779 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:20:50.945 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:20:53.670 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:20:53.670 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:20:53.670 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:20:53.670 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:20:53.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:20:53.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:20:55.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:20:58.596 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:21:01.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:21:03.516 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:21:05.424 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:21:08.529 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:21:09.701 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:21:11.835 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:23:13.908 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:23:15.365 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:23:18.401 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:23:21.373 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:23:23.556 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:23:23.556 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:23:23.556 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:23:23.556 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:23:23.556 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:23:23.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:23:23.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:23:26.557 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:23:31.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:23:33.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:23:36.632 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:23:38.328 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:23:39.513 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:23:40.814 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:23:41.664 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:23:43.489 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:25:45.092 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:25:47.004 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:25:49.450 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:25:51.804 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:25:53.401 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:25:53.402 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:25:53.402 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:25:53.402 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:25:53.402 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:25:53.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:25:53.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:25:55.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:25:58.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:26:01.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:26:02.553 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:26:04.770 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:26:06.550 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:26:07.806 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:26:07.807 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:26:07.900 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:26:09.413 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:28:10.973 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:28:11.603 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:28:13.001 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:28:14.212 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:28:16.426 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:28:17.638 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:28:17.639 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:28:17.639 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:28:17.639 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:28:17.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:28:18.324 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:28:21.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:28:22.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:28:23.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:28:24.182 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:28:25.459 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:28:27.065 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:28:29.818 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:28:29.818 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:28:29.859 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:28:31.983 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:30:36.684 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:30:37.981 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:30:38.948 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:30:39.921 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:30:41.211 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:30:41.211 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:30:41.211 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:30:41.211 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:30:41.211 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:30:41.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:30:41.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:30:42.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:30:43.650 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:30:45.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:30:46.796 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:30:48.000 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:30:48.694 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:30:49.550 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:30:49.919 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:30:49.966 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:30:50.830 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:32:52.130 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:32:53.210 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:32:53.924 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:32:54.989 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:32:57.587 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:32:57.587 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:32:57.587 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:32:57.587 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:32:57.587 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:32:57.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:32:57.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:32:59.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:33:01.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:33:02.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:33:09.228 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:33:10.749 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:33:12.355 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:33:13.373 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:33:13.373 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:33:13.421 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:33:14.813 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:35:16.699 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:35:17.808 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:35:18.715 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:35:21.170 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:35:22.573 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:35:22.574 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:35:22.574 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:35:22.574 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:35:22.574 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:35:22.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:35:22.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:35:24.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:35:25.491 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:35:26.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:35:27.486 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:35:27.538 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:35:28.012 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:35:28.517 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:35:28.518 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:35:28.559 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:35:29.595 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:37:30.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:37:32.318 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:37:33.657 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:37:34.692 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:37:35.889 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:37:35.890 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:37:35.890 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:37:37.863 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:37:37.863 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:37:37.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:37:38.046 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:37:39.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:37:40.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:37:41.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:37:42.844 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:37:43.899 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:37:45.911 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:37:46.727 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:37:46.727 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:37:46.806 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:37:48.038 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:39:49.184 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:39:52.433 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:39:54.817 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:39:57.674 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:39:58.882 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:39:58.882 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:39:58.882 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:39:58.882 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:39:58.883 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:39:58.883 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:40:01.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:40:03.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:40:05.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:40:06.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:40:07.075 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:40:07.655 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:40:08.882 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:40:10.263 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:40:10.264 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:40:10.264 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:40:11.526 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:40:12.827 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:42:13.786 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:42:15.566 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:42:17.824 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:42:19.679 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:42:21.507 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:42:21.507 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:42:21.507 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:42:21.507 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:42:21.507 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:42:21.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:42:21.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:42:23.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:42:26.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:42:27.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:42:30.359 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:42:31.366 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:42:32.529 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:42:34.147 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:42:34.147 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:42:34.153 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:42:34.195 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:42:34.966 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:44:35.807 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:44:36.529 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:44:37.502 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:44:38.490 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:44:39.446 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:44:39.446 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:44:39.446 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:44:39.446 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:44:39.446 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:44:39.446 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:44:39.623 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:44:40.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:44:42.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:44:43.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:44:44.832 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:44:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:44:47.048 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:44:48.599 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:44:50.075 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:44:50.076 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:44:50.811 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:44:52.286 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:46:53.768 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:46:55.452 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:46:58.333 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:47:00.369 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:47:02.402 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:47:02.402 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:47:02.403 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:47:02.403 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:47:02.403 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:47:03.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:47:04.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:47:05.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:47:07.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:47:08.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:47:10.984 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:47:11.650 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:47:13.003 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:47:13.004 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:47:13.004 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:47:13.004 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:47:13.045 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:47:14.472 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:49:15.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:49:16.708 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:49:17.590 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:49:18.841 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:49:21.695 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:49:21.696 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:49:21.696 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:49:21.696 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:49:21.696 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:49:23.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:49:24.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:49:27.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:49:29.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:49:30.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:49:34.350 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:49:36.702 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:49:38.212 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:49:40.009 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:49:40.009 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:49:40.009 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:49:40.057 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:49:41.603 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:51:42.715 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:51:45.037 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:51:46.768 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:51:48.660 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:51:50.254 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:51:51.500 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:51:51.500 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:51:51.500 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:51:51.500 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:51:51.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:51:51.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:51:52.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:51:54.852 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:51:57.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:52:00.026 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:52:02.439 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:52:05.157 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:52:07.599 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:52:07.600 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:52:07.600 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:52:07.602 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:52:08.988 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:52:10.588 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:54:12.449 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:54:13.300 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:54:16.610 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:54:17.268 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:54:19.896 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:54:19.896 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:54:19.896 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:54:19.896 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:54:19.896 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:54:19.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:54:20.075 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:54:23.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:54:24.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:54:25.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:54:28.312 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:54:29.350 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:54:31.590 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:54:31.590 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:54:31.590 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:54:31.590 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:54:31.591 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:54:31.669 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:54:32.905 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:56:34.936 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:56:36.383 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:56:37.578 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:56:38.381 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:56:39.791 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:56:39.792 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:56:39.792 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:56:39.792 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:56:39.792 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:56:39.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:56:39.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:56:43.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:56:45.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:56:47.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:56:48.672 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:56:49.865 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:56:51.877 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:56:51.877 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:56:51.877 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:56:51.877 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:56:51.877 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:56:54.022 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:56:56.767 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:58:59.580 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T03:59:01.307 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:59:02.737 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T03:59:04.565 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T03:59:06.638 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T03:59:06.638 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T03:59:06.638 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T03:59:06.638 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T03:59:06.638 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:59:06.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T03:59:06.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T03:59:08.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T03:59:11.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T03:59:13.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T03:59:26.532 INFO:teuthology.misc:Compressing logs...
2023-03-19T03:59:28.563 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:59:29.736 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:59:31.780 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:59:31.780 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:59:31.781 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T03:59:31.782 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T03:59:42.549 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T03:59:44.259 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:01:47.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:01:49.801 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:01:51.397 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:01:53.251 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:01:56.161 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:01:56.161 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:01:56.161 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:01:56.161 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:01:56.162 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:01:56.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:01:56.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:01:58.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:02:01.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:02:03.192 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:02:06.884 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:02:09.691 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:02:10.916 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:02:13.568 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:02:13.569 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:02:13.569 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:02:13.570 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:02:16.558 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:02:19.744 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:04:22.069 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:04:24.258 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:04:25.901 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:04:27.921 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:04:29.641 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:04:29.641 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:04:29.641 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:04:29.641 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:04:29.641 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:04:29.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:04:29.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:04:32.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:04:37.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:04:39.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:04:41.843 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:04:53.135 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:04:56.349 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:04:58.108 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:04:59.454 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:04:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:04:59.912 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:05:00.564 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:05:01.110 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:07:02.619 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:07:04.476 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:07:05.288 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:07:06.016 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:07:06.981 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:07:06.981 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:07:06.982 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:07:06.982 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:07:06.982 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:07:07.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:07:07.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:07:09.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:07:13.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:07:16.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:07:19.350 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:07:20.377 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:07:21.566 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:07:22.569 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:07:22.569 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:07:22.570 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:07:22.572 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:07:23.448 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:07:34.370 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:09:35.714 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:09:36.546 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:09:37.768 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:09:38.608 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:09:39.463 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:09:40.687 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:09:40.687 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:09:40.687 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:09:40.687 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:09:40.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:09:40.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:09:42.362 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:09:43.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:09:45.136 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:09:45.951 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:09:46.425 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:09:46.988 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:09:47.154 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:09:47.713 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:09:49.424 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:09:49.426 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:09:49.467 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:09:50.108 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:11:50.772 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:11:53.180 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:11:54.938 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:11:56.539 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:11:58.972 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:11:58.972 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:11:58.973 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:11:58.973 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:11:58.973 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:11:58.973 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:11:59.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:12:00.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:12:02.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:12:05.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:12:07.303 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:12:08.876 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:12:10.181 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:12:11.359 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:12:11.359 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:12:11.360 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:12:11.362 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:12:12.644 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:12:14.815 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:14:15.807 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:14:17.995 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:14:21.253 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:14:24.187 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:14:27.341 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:14:27.341 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:14:29.211 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:14:29.211 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:14:30.042 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:14:31.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:14:31.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:14:34.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:14:36.163 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:14:40.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:14:41.829 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:14:42.556 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:14:43.762 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:14:46.178 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:14:46.178 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:14:46.178 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:14:46.181 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:14:48.269 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:14:50.113 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:16:51.761 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:16:55.957 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:16:57.167 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:16:58.471 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:17:01.274 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:17:01.274 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:17:01.274 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:17:01.275 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:17:01.275 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:17:01.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:17:01.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:17:03.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:17:04.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:17:05.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:17:08.198 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:17:09.626 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:17:11.374 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:17:13.835 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:17:13.836 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:17:13.836 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:17:13.837 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:17:13.877 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:17:15.898 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:19:18.169 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:19:19.092 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:19:20.453 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:19:22.311 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:19:23.725 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:19:23.725 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:19:23.725 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:19:23.725 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:19:23.726 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:19:23.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:19:23.860 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:19:25.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:19:28.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:19:30.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:19:31.667 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:19:33.417 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:19:34.996 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:19:36.206 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:19:38.085 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:19:39.127 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:19:39.128 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:19:39.168 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:19:40.363 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:21:40.762 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:21:42.461 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:21:44.762 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:21:46.021 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:21:47.011 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:21:47.011 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:21:47.011 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:21:47.011 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:21:47.011 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:21:47.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:21:47.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:21:49.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:21:51.925 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:21:53.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:21:55.249 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:21:56.977 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:21:58.790 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:22:00.311 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:22:00.725 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:22:01.501 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:22:01.502 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:22:02.853 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:22:04.944 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:24:06.119 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:24:07.187 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:24:08.527 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:24:09.708 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:24:10.774 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:24:10.774 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:24:10.774 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:24:10.774 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:24:11.116 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:24:11.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:24:11.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:24:13.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:24:14.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:24:16.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:24:18.434 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:24:19.269 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:24:19.527 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:24:20.705 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:24:22.364 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:24:22.364 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:24:22.367 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:24:24.144 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:24:25.981 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:26:27.571 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:26:28.583 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:26:30.278 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:26:31.613 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:26:32.513 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:26:32.513 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:26:32.513 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:26:32.513 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:26:32.513 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:26:32.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:26:32.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:26:33.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:26:34.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:26:35.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:26:37.029 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:26:37.746 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:26:39.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:26:39.583 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:26:39.583 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:26:39.584 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:26:39.584 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:26:39.623 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:26:41.367 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:28:42.842 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:28:45.689 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:28:47.410 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:28:49.952 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:28:52.985 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:28:52.985 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:28:52.985 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:28:52.985 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:28:52.986 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:28:52.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:28:53.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:28:54.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:28:56.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:28:57.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:28:59.829 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:29:00.545 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:29:02.028 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:29:02.028 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:29:02.028 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:29:02.028 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:29:02.029 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:29:02.068 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:29:03.551 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:31:04.413 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:31:05.355 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:31:06.838 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:31:09.460 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:31:12.051 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:31:12.051 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:31:12.051 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:31:12.051 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:31:13.684 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:31:13.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:31:13.864 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:31:15.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:31:17.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:31:18.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:31:21.130 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:31:24.334 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:31:25.682 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:31:27.093 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:31:27.093 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:31:27.093 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:31:28.494 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:31:28.534 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:31:29.689 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:33:30.255 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:33:33.093 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:33:36.050 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:33:38.176 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:33:38.778 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:33:38.778 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:33:38.778 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:33:38.778 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:33:38.778 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:33:38.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:33:39.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:33:40.502 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:33:41.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:33:42.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:33:44.304 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:33:45.825 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:33:47.583 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:33:47.583 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:33:47.583 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:33:47.584 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:33:47.584 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:33:48.664 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:33:49.692 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:35:50.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:35:53.735 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:35:55.244 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:35:57.635 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:35:58.731 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:35:58.731 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:35:58.732 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:35:58.732 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:35:58.732 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:35:58.732 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:35:58.912 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:36:00.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:36:01.353 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:36:02.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:36:04.366 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:36:05.437 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:36:06.415 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:36:07.502 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:36:07.502 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:36:07.502 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:36:07.504 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:36:10.654 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:36:15.021 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:38:17.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:38:19.834 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:38:21.579 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:38:24.572 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:38:26.005 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:38:26.005 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:38:26.005 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:38:26.005 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:38:26.005 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:38:26.006 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:38:26.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:38:28.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:38:30.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:38:32.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:38:34.073 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:38:34.812 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:38:35.535 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:38:36.553 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:38:36.553 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:38:36.554 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:38:36.556 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:38:36.600 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:38:37.992 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:40:39.487 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:40:41.176 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:40:42.700 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:40:43.790 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:40:44.448 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:40:44.448 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:40:44.448 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:40:44.448 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:40:44.449 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:40:44.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:40:44.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:40:46.017 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:40:47.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:40:49.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:40:52.156 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:40:53.639 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:40:56.306 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:40:58.427 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:40:58.427 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:40:58.428 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:40:58.430 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:41:00.150 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:41:01.968 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:43:03.032 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:43:04.192 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:43:06.181 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:43:07.939 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:43:08.677 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:43:08.677 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:43:08.677 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:43:08.677 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:43:08.678 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:43:08.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:43:08.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:43:10.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:43:11.825 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:43:13.549 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:43:16.093 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:43:17.728 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:43:20.239 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:43:20.239 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:43:20.239 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:43:20.240 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:43:20.240 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:43:20.279 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:43:22.497 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:45:24.777 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:45:26.345 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:45:27.636 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:45:29.829 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:45:30.536 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:45:32.095 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:45:32.095 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:45:32.095 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:45:32.095 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:45:32.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:45:32.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:45:34.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:45:35.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:45:36.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:45:37.878 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:45:39.199 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:45:40.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:45:45.591 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:45:45.591 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:45:45.591 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:45:45.592 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:45:45.631 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:45:48.586 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:47:49.555 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:47:50.345 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:47:51.984 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:47:53.474 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:47:54.737 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:47:54.737 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:47:54.738 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:47:54.738 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:47:54.738 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:47:54.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:47:54.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:47:56.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:47:57.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:47:58.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:47:59.818 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:48:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:48:02.281 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:48:05.193 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:48:05.194 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:48:05.194 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:48:05.828 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:48:07.130 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:48:10.820 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:50:10.890 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:50:11.572 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:50:13.026 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:50:14.388 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:50:15.783 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:50:15.784 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:50:15.784 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:50:15.784 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:50:17.799 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:50:17.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:50:17.978 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:50:23.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:50:27.550 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:50:30.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:50:33.079 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:50:34.623 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:50:37.665 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:50:39.327 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:50:39.327 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:50:40.286 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:50:40.288 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:50:43.581 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:50:44.751 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:52:46.116 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:52:47.812 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:52:48.839 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:52:52.518 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:52:55.695 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:52:55.695 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:52:55.695 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:52:55.695 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:52:55.695 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:52:55.695 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:52:55.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:53:00.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:53:04.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:53:05.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:53:07.707 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:53:09.086 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:53:10.202 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:53:11.711 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:53:11.711 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:53:11.711 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:53:11.713 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:53:11.797 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:53:15.550 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:55:16.832 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:55:19.199 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:55:22.544 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:55:24.605 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:55:24.999 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:55:25.984 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:55:25.984 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:55:25.984 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:55:25.984 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:55:25.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:55:27.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:55:28.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:55:29.464 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:55:31.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:55:34.702 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:55:36.920 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:55:38.108 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:55:40.236 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:55:40.236 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:55:40.236 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:55:40.239 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:55:41.600 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:55:42.683 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:57:45.492 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T04:57:46.633 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:57:47.801 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T04:57:49.534 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T04:57:51.526 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T04:57:51.527 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T04:57:52.329 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T04:57:52.329 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T04:57:52.329 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:57:52.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T04:57:52.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T04:57:53.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T04:57:54.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T04:57:57.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T04:57:59.182 INFO:teuthology.misc:Compressing logs...
2023-03-19T04:58:00.251 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:58:01.973 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:58:04.410 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:58:04.410 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:58:04.410 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T04:58:05.979 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T04:58:06.021 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T04:58:07.894 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:00:09.519 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:00:10.406 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:00:12.561 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:00:14.849 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:00:16.561 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:00:16.561 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:00:16.561 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:00:16.561 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:00:16.562 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:00:16.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:00:16.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:00:19.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:00:21.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:00:23.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:00:24.888 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:00:25.544 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:00:26.761 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:00:27.856 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:00:29.239 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:00:30.249 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:00:30.252 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:00:30.294 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:00:31.194 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:02:31.681 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:02:33.738 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:02:36.298 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:02:37.223 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:02:38.668 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:02:38.668 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:02:38.668 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:02:38.668 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:02:38.668 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:02:38.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:02:38.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:02:40.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:02:42.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:02:44.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:02:46.181 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:02:46.755 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:02:47.523 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:02:48.281 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:02:48.745 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:02:49.458 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:02:49.460 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:02:49.924 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:02:51.060 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:04:53.543 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:04:55.202 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:04:57.015 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:04:59.355 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:05:02.469 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:05:02.469 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:05:02.469 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:05:03.338 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:05:03.338 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:05:04.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:05:04.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:05:08.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:05:12.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:05:13.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:05:16.342 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:05:17.672 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:05:20.109 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:05:21.021 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:05:21.021 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:05:21.022 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:05:21.025 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:05:22.927 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:05:23.317 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:07:25.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:07:26.770 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:07:28.080 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:07:29.606 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:07:30.379 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:07:30.379 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:07:30.379 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:07:30.379 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:07:30.379 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:07:30.379 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:07:31.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:07:32.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:07:33.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:07:33.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:07:35.453 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:07:37.917 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:07:40.846 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:07:44.125 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:07:44.125 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:07:44.125 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:07:44.127 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:07:44.168 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:07:45.786 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:09:48.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:09:50.963 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:09:52.502 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:09:53.466 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:09:56.129 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:09:56.129 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:09:56.129 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:09:56.130 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:09:56.130 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:09:56.130 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:09:56.326 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:09:58.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:10:00.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:10:01.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:10:02.744 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:10:03.419 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:10:03.788 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:10:04.359 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:10:04.359 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:10:04.359 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:10:04.361 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:10:04.402 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:10:05.047 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:12:05.114 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:12:07.120 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:12:09.552 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:12:11.519 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:12:15.156 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:12:15.156 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:12:15.156 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:12:15.156 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:12:15.156 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:12:15.156 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:12:15.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:12:16.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:12:19.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:12:20.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:12:22.434 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:12:24.721 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:12:27.501 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:12:33.732 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:12:33.733 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:12:35.727 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:12:35.729 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:12:35.775 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:12:38.199 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:14:42.665 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:14:43.753 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:14:45.778 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:14:47.690 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:14:50.772 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:14:50.772 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:14:50.772 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:14:50.772 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:14:50.772 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:14:50.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:14:50.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:14:53.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:14:54.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:14:56.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:14:58.669 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:14:59.978 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:15:02.031 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:15:02.031 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:15:02.031 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:15:02.031 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:15:02.032 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:15:02.074 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:15:03.025 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:17:04.113 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:17:06.941 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:17:09.178 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:17:11.449 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:17:14.163 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:17:14.163 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:17:14.163 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:17:14.163 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:17:14.163 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:17:14.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:17:14.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:17:17.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:17:20.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:17:22.056 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:17:24.677 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:17:25.674 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:17:27.080 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:17:27.081 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:17:27.081 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:17:27.081 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:17:29.186 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:17:29.228 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:17:31.040 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:19:32.914 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:19:33.524 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:19:36.015 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:19:37.136 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:19:40.113 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:19:40.113 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:19:40.113 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:19:40.113 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:19:40.113 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:19:40.113 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:19:40.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:19:41.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:19:43.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:19:45.444 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:19:47.746 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:19:48.751 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:19:50.161 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:19:51.280 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:19:51.280 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:19:51.280 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:19:51.281 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:19:51.320 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:19:52.384 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:21:53.432 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:21:55.113 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:21:56.722 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:21:58.638 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:22:00.640 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:22:00.640 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:22:00.640 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:22:00.640 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:22:01.842 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:22:01.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:22:03.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:22:04.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:22:05.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:22:06.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:22:08.722 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:22:08.964 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:22:10.277 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:22:11.745 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:22:11.745 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:22:11.745 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:22:11.747 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:22:13.341 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:22:14.457 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:24:16.520 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:24:18.083 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:24:19.696 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:24:20.722 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:24:21.547 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:24:22.475 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:24:22.475 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:24:23.647 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:24:23.648 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:24:23.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:24:23.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:24:24.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:24:25.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:24:26.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:24:28.295 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:24:30.148 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:24:31.972 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:24:33.263 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:24:33.263 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:24:33.263 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:24:33.265 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:24:33.311 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:24:34.527 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:26:35.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:26:37.969 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:26:39.003 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:26:41.046 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:26:44.050 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:26:44.050 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:26:44.050 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:26:44.050 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:26:44.050 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:26:44.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:26:46.087 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:26:47.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:26:48.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:26:51.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:26:53.174 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:26:54.914 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:26:57.248 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:27:00.060 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:27:00.060 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:27:00.061 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:27:00.062 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:27:01.287 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:27:01.994 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:29:03.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:29:04.788 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:29:05.707 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:29:06.943 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:29:07.931 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:29:07.931 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:29:08.781 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:29:08.781 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:29:08.781 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:29:08.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:29:08.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:29:10.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:29:12.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:29:13.962 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:29:16.580 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:29:18.452 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:29:20.649 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:29:20.650 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:29:20.650 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:29:22.037 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:29:22.709 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:29:22.750 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:29:23.726 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:31:25.770 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:31:29.310 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:31:32.553 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:31:33.918 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:31:35.279 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:31:35.279 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:31:37.343 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:31:37.962 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:31:37.963 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:31:37.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:31:38.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:31:40.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:31:42.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:31:43.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:31:44.457 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:31:45.947 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:31:47.436 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:31:49.122 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:31:50.129 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:31:50.129 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:31:50.854 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:31:52.388 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:31:55.439 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:33:57.137 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:33:58.496 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:33:59.897 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:34:01.381 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:34:02.634 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:34:02.634 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:34:02.634 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:34:02.634 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:34:02.634 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:34:02.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:34:02.817 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:34:04.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:34:06.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:34:08.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:34:09.213 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:34:09.814 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:34:11.402 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:34:12.748 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:34:12.749 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:34:13.570 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:34:14.280 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:34:15.765 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:34:17.467 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:36:18.989 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:36:20.560 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:36:21.525 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:36:22.381 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:36:22.709 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:36:22.709 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:36:22.709 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:36:22.709 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:36:22.709 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:36:22.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:36:23.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:36:24.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:36:24.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:36:25.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:36:26.283 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:36:26.544 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:36:26.920 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:36:27.551 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:36:27.551 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:36:27.551 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:36:27.554 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:36:27.777 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:36:28.285 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:38:28.519 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:38:29.663 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:38:32.301 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:38:35.143 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:38:37.990 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:38:37.990 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:38:37.990 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:38:37.990 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:38:37.990 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:38:37.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:38:38.180 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:38:39.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:38:41.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:38:43.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:38:44.222 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:38:45.276 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:38:47.800 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:38:47.800 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:38:47.800 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:38:47.800 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:38:47.800 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:38:47.847 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:38:49.101 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:40:51.181 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:40:52.174 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:40:54.063 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:40:55.319 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:40:57.276 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:40:57.276 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:40:58.497 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:40:58.497 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:40:58.498 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:40:58.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:40:59.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:41:00.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:41:01.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:41:03.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:41:04.777 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:41:05.538 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:41:07.641 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:41:07.641 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:41:07.641 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:41:07.641 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:41:07.642 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:41:09.790 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:41:13.772 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:43:14.717 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:43:15.946 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:43:18.596 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:43:19.987 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:43:21.814 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:43:21.814 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:43:21.814 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:43:21.815 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:43:23.354 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:43:23.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:43:23.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:43:24.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:43:25.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:43:27.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:43:29.317 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:43:29.929 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:43:31.117 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:43:32.420 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:43:32.420 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:43:32.420 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:43:32.422 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:43:32.464 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:43:33.573 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:45:34.486 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:45:38.277 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:45:40.386 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:45:43.383 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:45:45.203 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:45:45.203 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:45:45.203 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:45:45.203 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:45:45.204 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:45:45.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:45:45.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:45:47.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:45:49.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:45:51.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:45:53.564 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:45:54.781 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:45:56.270 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:45:59.811 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:45:59.811 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:45:59.811 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:45:59.813 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:46:00.820 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:46:02.625 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:48:04.607 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:48:05.755 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:48:07.320 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:48:08.631 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:48:09.965 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:48:09.966 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:48:09.966 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:48:09.966 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:48:09.966 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:48:11.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:48:11.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:48:13.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:48:15.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:48:16.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:48:18.024 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:48:19.437 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:48:20.859 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:48:24.823 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:48:24.823 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:48:24.823 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:48:25.480 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:48:25.522 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:48:26.662 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:50:28.267 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:50:29.096 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:50:30.707 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:50:32.775 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:50:33.917 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:50:33.917 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:50:33.917 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:50:33.917 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:50:33.917 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:50:33.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:50:35.465 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:50:36.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:50:38.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:50:40.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:50:42.551 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:50:44.446 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:50:45.694 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:50:45.694 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:50:47.432 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:50:47.432 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:50:47.434 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:50:47.525 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:50:48.622 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:52:50.355 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:52:52.243 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:52:53.227 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:52:54.141 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:52:55.847 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:52:55.847 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:52:55.847 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:52:55.848 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:52:55.848 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:52:55.848 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:52:55.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:52:57.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:53:01.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:53:04.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:53:06.034 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:53:07.131 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:53:08.117 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:53:08.117 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:53:08.117 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:53:08.117 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:53:08.117 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:53:08.158 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:53:09.641 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:55:11.532 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:55:15.359 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:55:17.834 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:55:20.112 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:55:22.657 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:55:22.657 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:55:22.657 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:55:22.657 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:55:22.657 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:55:22.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:55:22.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:55:25.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:55:25.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:55:28.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:55:30.532 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:55:31.974 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:55:32.988 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:55:35.441 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:55:35.441 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:55:37.344 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:55:37.346 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:55:37.388 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:55:38.787 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:57:40.221 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T05:57:43.358 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:57:44.003 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:57:45.729 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T05:57:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T05:57:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T05:57:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T05:57:47.521 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T05:57:47.521 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:57:47.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T05:57:47.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T05:57:49.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T05:57:51.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T05:57:53.219 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T05:57:54.695 INFO:teuthology.misc:Compressing logs...
2023-03-19T05:57:55.730 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:57:56.564 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:57:57.500 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:57:57.500 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:57:57.500 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T05:57:57.502 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T05:57:57.542 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T05:57:58.231 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T05:59:59.664 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:00:01.738 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:00:03.139 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:00:06.200 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:00:08.647 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:00:08.648 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:00:08.648 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:00:10.044 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:00:10.044 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:00:10.044 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:00:10.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:00:11.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:00:12.803 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:00:15.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:00:17.218 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:00:19.065 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:00:19.467 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:00:20.312 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:00:20.313 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:00:20.313 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:00:20.316 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:00:22.251 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:00:23.585 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:02:24.569 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:02:25.648 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:02:28.141 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:02:30.625 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:02:31.692 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:02:31.692 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:02:32.033 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:02:32.600 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:02:32.600 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:02:32.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:02:32.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:02:34.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:02:35.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:02:37.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:02:38.781 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:02:40.035 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:02:41.435 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:02:42.679 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:02:42.679 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:02:42.680 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:02:42.681 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:02:42.720 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:02:44.535 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:04:46.080 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:04:47.016 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:04:49.620 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:04:51.478 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:04:53.204 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:04:53.205 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:04:53.205 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:04:53.205 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:04:53.205 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:04:53.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:04:53.389 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:04:54.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:04:55.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:04:56.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:04:57.435 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:04:58.274 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:04:59.495 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:04:59.495 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:04:59.496 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:04:59.496 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:04:59.496 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:04:59.535 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:05:02.595 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:07:05.452 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:07:06.859 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:07:09.710 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:07:11.532 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:07:13.042 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:07:13.042 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:07:13.657 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:07:13.657 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:07:13.657 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:07:13.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:07:13.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:07:16.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:07:19.372 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:07:22.401 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:07:23.928 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:07:27.390 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:07:28.847 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:07:30.380 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:07:30.380 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:07:30.381 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:07:30.382 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:07:30.422 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:07:32.481 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:09:36.595 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:09:37.521 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:09:38.438 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:09:40.004 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:09:41.868 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:09:41.868 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:09:41.868 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:09:41.868 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:09:41.869 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:09:41.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:09:42.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:09:45.760 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:09:47.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:09:51.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:09:55.158 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:09:56.299 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:09:57.374 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:09:58.928 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:10:00.781 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:10:00.781 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:10:00.783 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:10:03.922 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:10:05.374 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:12:07.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:12:08.140 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:12:10.625 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:12:11.891 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:12:13.980 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:12:13.980 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:12:13.980 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:12:13.980 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:12:13.980 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:12:13.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:12:14.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:12:15.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:12:17.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:12:19.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:12:25.427 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:12:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:12:27.483 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:12:28.130 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:12:28.130 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:12:28.130 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:12:29.408 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:12:29.448 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:12:31.195 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:14:34.620 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:14:39.543 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:14:41.119 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:14:43.555 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:14:44.133 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:14:44.133 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:14:44.134 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:14:44.134 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:14:44.134 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:14:44.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:14:44.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:14:45.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:14:45.632 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:14:46.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:14:47.574 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:14:48.977 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:14:49.591 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:14:52.652 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:14:52.652 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:14:52.652 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:14:52.655 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:14:53.738 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:14:55.299 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:16:58.616 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:17:01.499 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:17:05.020 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:17:08.630 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:17:12.225 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:17:12.226 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:17:12.226 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:17:12.226 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:17:12.226 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:17:12.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:17:12.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:17:15.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:17:17.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:17:19.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:17:21.863 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:17:23.310 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:17:24.120 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:17:25.295 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:17:25.295 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:17:25.297 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:17:25.970 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:17:26.012 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:17:27.748 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:19:28.575 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:19:29.522 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:19:29.830 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:19:30.940 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:19:31.934 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:19:31.934 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:19:31.934 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:19:31.934 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:19:31.934 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:19:31.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:19:32.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:19:33.248 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:19:34.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:19:35.745 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:19:38.450 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:19:39.174 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:19:39.709 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:19:40.771 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:19:40.771 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:19:40.771 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:19:40.773 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:19:42.852 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:19:43.327 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:21:44.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:21:45.774 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:21:47.730 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:21:49.668 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:21:51.879 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:21:51.879 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:21:51.879 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:21:51.879 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:21:51.879 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:21:51.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:21:52.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:21:55.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:21:56.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:21:57.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:22:00.603 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:22:02.486 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:22:04.706 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:22:07.767 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:22:07.767 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:22:07.767 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:22:07.769 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:22:10.339 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:22:12.577 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:24:15.207 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:24:18.441 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:24:21.482 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:24:22.490 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:24:23.796 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:24:23.797 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:24:23.797 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:24:23.797 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:24:23.797 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:24:23.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:24:23.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:24:24.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:24:25.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:24:26.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:24:28.447 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:24:29.673 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:24:30.527 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:24:31.831 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:24:31.831 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:24:31.831 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:24:31.832 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:24:31.881 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:24:32.683 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:26:33.453 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:26:34.091 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:26:35.737 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:26:36.806 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:26:37.846 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:26:37.846 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:26:37.846 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:26:37.846 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:26:37.846 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:26:37.846 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:26:38.037 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:26:39.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:26:41.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:26:43.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:26:45.278 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:26:47.183 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:26:50.485 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:26:51.717 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:26:51.717 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:26:51.717 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:26:51.719 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:26:51.760 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:26:53.577 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:28:55.573 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:28:58.250 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:29:01.972 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:29:03.668 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:29:05.544 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:29:05.544 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:29:08.691 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:29:08.691 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:29:09.936 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:29:09.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:29:10.122 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:29:11.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:29:13.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:29:14.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:29:15.882 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:29:17.073 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:29:17.716 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:29:19.116 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:29:19.117 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:29:19.117 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:29:19.118 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:29:20.264 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:29:20.777 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:31:21.382 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:31:22.625 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:31:23.206 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:31:24.041 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:31:25.346 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:31:25.346 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:31:25.346 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:31:25.346 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:31:25.346 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:31:25.346 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:31:25.533 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:31:26.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:31:27.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:31:29.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:31:31.442 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:31:32.715 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:31:34.502 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:31:37.270 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:31:37.920 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:31:37.920 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:31:37.922 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:31:40.012 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:31:41.308 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:33:43.712 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:33:44.759 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:33:45.746 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:33:47.986 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:33:49.496 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:33:50.699 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:33:51.506 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:33:51.506 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:33:51.506 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:33:51.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:33:51.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:33:53.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:33:55.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:33:57.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:33:58.919 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:34:00.075 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:34:02.104 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:34:04.153 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:34:04.153 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:34:04.153 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:34:04.155 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:34:06.135 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:34:07.728 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:36:08.954 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:36:10.593 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:36:11.981 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:36:13.383 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:36:17.086 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:36:17.087 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:36:17.946 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:36:17.946 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:36:17.946 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:36:17.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:36:19.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:36:22.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:36:23.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:36:24.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:36:26.244 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:36:27.773 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:36:30.680 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:36:32.030 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:36:33.918 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:36:35.774 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:36:35.775 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:36:38.781 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:36:41.715 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:38:43.198 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:38:45.657 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:38:48.319 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:38:50.795 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:38:52.363 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:38:52.363 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:38:52.364 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:38:52.364 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:38:52.364 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:38:52.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:38:52.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:38:55.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:38:58.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:39:00.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:39:03.005 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:39:05.883 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:39:07.078 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:39:08.103 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:39:09.100 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:39:09.100 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:39:09.814 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:39:12.060 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:39:14.599 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:41:16.406 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:41:17.294 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:41:19.320 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:41:22.371 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:41:23.614 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:41:23.614 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:41:23.614 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:41:23.614 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:41:23.614 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:41:23.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:41:23.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:41:25.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:41:28.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:41:29.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:41:31.085 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:41:31.892 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:41:33.368 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:41:35.020 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:41:35.020 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:41:35.020 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:41:35.022 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:41:35.063 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:41:38.634 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:43:40.166 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:43:41.105 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:43:42.413 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:43:44.274 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:43:47.324 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:43:49.678 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:43:49.678 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:43:49.678 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:43:49.679 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:43:49.679 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:43:49.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:43:52.489 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:43:54.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:43:57.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:44:00.462 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:44:01.866 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:44:03.471 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:44:04.873 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:44:04.874 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:44:04.874 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:44:04.875 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:44:04.917 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:44:06.180 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:46:08.222 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:46:09.492 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:46:11.022 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:46:11.758 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:46:13.316 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:46:13.316 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:46:13.316 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:46:13.316 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:46:13.317 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:46:13.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:46:13.496 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:46:14.415 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:46:15.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:46:17.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:46:19.250 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:46:20.884 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:46:22.488 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:46:24.301 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:46:24.301 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:46:24.302 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:46:24.303 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:46:24.342 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:46:25.432 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:48:27.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:48:27.850 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:48:28.878 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:48:30.579 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:48:33.584 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:48:33.584 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:48:33.584 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:48:33.584 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:48:33.585 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:48:33.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:48:33.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:48:35.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:48:36.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:48:38.144 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:48:40.490 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:48:40.757 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:48:42.349 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:48:43.183 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:48:43.183 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:48:43.388 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:48:43.392 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:48:43.876 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:48:44.656 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:50:45.053 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:50:46.931 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:50:48.203 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:50:49.597 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:50:50.763 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:50:50.763 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:50:50.763 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:50:50.763 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:50:50.763 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:50:50.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:50:50.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:50:53.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:50:55.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:50:57.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:50:58.535 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:50:59.184 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:51:00.361 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:51:00.361 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:51:00.362 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:51:00.362 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:51:00.362 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:51:00.402 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:51:01.299 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:53:02.501 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:53:03.730 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:53:04.723 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:53:05.761 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:53:08.337 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:53:08.337 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:53:08.337 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:53:08.337 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:53:08.337 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:53:08.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:53:10.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:53:13.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:53:15.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:53:16.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:53:18.519 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:53:20.389 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:53:21.372 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:53:22.205 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:53:22.205 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:53:22.205 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:53:22.207 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:53:22.253 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:53:23.598 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:55:24.556 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:55:25.190 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:55:25.872 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:55:26.630 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:55:27.870 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:55:27.870 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:55:27.870 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:55:27.871 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:55:28.243 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:55:28.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:55:28.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:55:29.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:55:29.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:55:31.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:55:32.924 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:55:35.574 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:55:38.729 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:55:40.562 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:55:40.562 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:55:40.563 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:55:40.564 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:55:40.616 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:55:44.196 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:57:45.441 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T06:57:46.737 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:57:49.831 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T06:57:51.890 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T06:57:54.687 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T06:57:54.687 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T06:57:54.687 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T06:57:54.687 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T06:57:54.688 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:57:54.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T06:57:54.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T06:57:56.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T06:57:57.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T06:58:00.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T06:58:01.904 INFO:teuthology.misc:Compressing logs...
2023-03-19T06:58:02.514 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:58:04.043 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:58:05.033 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:58:05.033 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:58:05.033 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T06:58:05.037 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T06:58:06.670 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T06:58:08.725 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:00:10.017 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:00:11.844 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:00:14.051 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:00:16.346 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:00:18.064 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:00:18.064 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:00:18.064 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:00:18.064 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:00:18.064 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:00:18.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:00:18.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:00:23.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:00:26.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:00:27.889 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:00:28.967 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:00:30.055 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:00:30.785 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:00:31.978 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:00:31.978 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:00:31.978 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:00:31.979 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:00:33.227 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:00:34.524 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:02:35.478 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:02:36.895 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:02:39.393 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:02:41.454 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:02:43.047 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:02:43.047 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:02:43.047 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:02:43.047 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:02:43.048 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:02:43.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:02:43.232 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:02:45.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:02:46.553 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:02:49.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:02:52.142 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:02:53.685 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:02:55.740 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:02:56.672 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:02:56.673 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:02:56.673 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:02:56.675 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:02:58.620 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:03:01.490 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:05:04.404 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:05:05.545 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:05:06.449 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:05:07.508 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:05:08.493 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:05:08.493 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:05:08.494 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:05:08.494 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:05:09.644 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:05:09.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:05:09.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:05:12.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:05:14.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:05:17.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:05:20.065 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:05:22.114 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:05:24.064 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:05:25.220 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:05:25.221 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:05:27.116 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:05:29.655 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:05:29.695 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:05:33.908 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:07:36.348 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:07:37.655 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:07:39.214 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:07:40.957 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:07:42.705 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:07:42.706 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:07:42.706 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:07:42.706 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:07:42.706 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:07:42.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:07:42.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:07:44.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:07:48.254 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:07:53.239 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:07:54.672 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:07:55.673 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:07:57.917 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:07:59.302 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:07:59.302 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:08:00.413 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:08:00.415 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:08:02.642 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:08:05.067 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:10:07.339 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:10:08.578 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:10:12.012 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:10:13.663 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:10:15.332 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:10:15.332 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:10:16.706 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:10:16.706 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:10:16.706 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:10:16.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:10:16.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:10:18.542 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:10:19.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:10:21.293 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:10:23.721 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:10:25.513 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:10:27.347 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:10:28.872 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:10:28.872 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:10:28.872 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:10:28.874 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:10:28.915 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:10:30.368 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:12:33.130 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:12:36.875 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:12:37.940 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:12:41.529 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:12:43.322 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:12:43.322 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:12:43.322 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:12:43.322 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:12:43.322 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:12:43.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:12:43.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:12:45.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:12:48.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:12:50.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:12:54.450 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:12:56.644 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:12:59.327 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:13:01.577 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:13:01.577 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:13:01.577 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:13:01.579 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:13:04.071 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:13:05.364 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:15:07.251 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:15:10.636 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:15:13.061 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:15:14.608 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:15:16.547 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:15:18.045 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:15:18.045 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:15:18.045 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:15:18.045 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:15:18.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:15:18.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:15:20.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:15:22.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:15:24.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:15:26.490 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:15:28.854 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:15:30.406 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:15:31.521 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:15:31.521 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:15:31.521 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:15:31.523 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:15:31.564 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:15:32.682 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:17:33.671 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:17:34.988 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:17:36.326 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:17:37.213 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:17:38.716 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:17:38.716 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:17:38.716 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:17:38.716 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:17:38.716 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:17:38.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:17:38.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:17:41.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:17:43.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:17:44.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:17:47.127 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:17:48.688 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:17:51.142 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:17:54.017 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:17:54.017 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:17:54.018 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:17:54.019 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:17:54.098 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:17:55.711 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:19:57.922 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-03-19T07:19:58.794 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:20:00.485 ERROR:teuthology.dispatcher.supervisor:Failed to kill job
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:20:02.613 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0']
2023-03-19T07:20:04.126 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1']
2023-03-19T07:20:04.126 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2']
2023-03-19T07:20:04.126 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3']
2023-03-19T07:20:04.126 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4']
2023-03-19T07:20:04.127 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:20:04.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com'
2023-03-19T07:20:04.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2023-03-19T07:20:06.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com'
2023-03-19T07:20:08.384 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com'
2023-03-19T07:20:10.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi188.front.sepia.ceph.com'
2023-03-19T07:20:12.235 INFO:teuthology.misc:Compressing logs...
2023-03-19T07:20:13.584 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:20:15.422 INFO:teuthology.orchestra.run.smithi188.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:20:18.572 INFO:teuthology.orchestra.run.smithi187.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:20:18.572 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:20:18.573 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log.gz already exists;	not overwritten
2023-03-19T07:20:18.574 ERROR:teuthology.dispatcher.supervisor:Could not save logs
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog
    transfer_archives(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives
    compress_logs(ctx, log_path)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/misc.py", line 1367, in compress_logs
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi027 with status 123: 'sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --'
2023-03-19T07:20:20.421 INFO:teuthology.kill:Killing Pids: {26761}
2023-03-19T07:20:21.901 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog
    kill_job(job_info['name'], job_info['job_id'],
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/kill.py", line 90, in kill_job
    teuthology.exporter.JobResults().record(job_info["machine_type"], job_info["status"])
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/teuthology/exporter.py", line 175, in __init__
    self.metric = Counter(
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__
    registry.register(self)
  File "/home/teuthworker/src/git.ceph.com_teuthology_5c9057f4b1ecae7b7410e418e40b739ff0616d25/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register
    raise ValueError(
ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results', 'teuthology_job_results_created', 'teuthology_job_results_total'}
2023-03-19T07:22:23.615 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-03-19T07:22:25.370 INFO:teuthology.dispatcher.supervisor:Nuking machines...
2023-03-19T07:22:26.926 INFO:teuthology.nuke:Checking targets against current locks
2023-03-19T07:22:30.363 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T07:22:31.425 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T07:22:33.382 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T07:22:34.848 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T07:22:36.495 INFO:teuthology.orchestra.console:Power off smithi027
2023-03-19T07:22:37.216 INFO:teuthology.orchestra.console:Power off smithi037
2023-03-19T07:22:38.566 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-03-19T07:22:39.992 INFO:teuthology.orchestra.console:Power off smithi103
2023-03-19T07:22:41.324 INFO:teuthology.orchestra.console:Power off smithi187
2023-03-19T07:22:43.845 INFO:teuthology.orchestra.console:Power off smithi188
2023-03-19T07:22:46.909 INFO:teuthology.orchestra.console:Power off for smithi027 completed
2023-03-19T07:22:48.585 INFO:teuthology.lock.ops:unlocked: smithi027.front.sepia.ceph.com
2023-03-19T07:22:51.409 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2023-03-19T07:22:51.567 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com
2023-03-19T07:22:54.430 INFO:teuthology.orchestra.console:Power off for smithi103 completed
2023-03-19T07:22:56.238 INFO:teuthology.orchestra.console:Power off for smithi187 completed
2023-03-19T07:22:56.883 INFO:teuthology.lock.ops:unlocked: smithi103.front.sepia.ceph.com
2023-03-19T07:22:58.433 INFO:teuthology.lock.ops:unlocked: smithi187.front.sepia.ceph.com
2023-03-19T07:23:01.158 INFO:teuthology.orchestra.console:Power off for smithi188 completed
2023-03-19T07:23:02.241 INFO:teuthology.lock.ops:unlocked: smithi188.front.sepia.ceph.com