2023-03-19T07:13:02.210 INFO:root:teuthology version: 0.0.1.dev59+g7f7f7c3 2023-03-19T07:13:05.114 INFO:teuthology.lock.ops:Start node 'smithi072.front.sepia.ceph.com' reimaging 2023-03-19T07:13:07.183 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: reset os type and version on server 2023-03-19T07:13:07.183 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-03-19T07:13:07.215 INFO:teuthology.lock.ops:Node 'smithi072.front.sepia.ceph.com' reimaging is complete 2023-03-19T07:13:12.292 INFO:teuthology.lock.ops:Start node 'smithi112.front.sepia.ceph.com' reimaging 2023-03-19T07:13:12.292 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: reset os type and version on server 2023-03-19T07:13:12.293 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2023-03-19T07:13:12.322 INFO:teuthology.lock.ops:Node 'smithi112.front.sepia.ceph.com' reimaging is complete 2023-03-19T07:13:13.859 INFO:teuthology.lock.ops:Start node 'smithi122.front.sepia.ceph.com' reimaging 2023-03-19T07:13:13.859 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: reset os type and version on server 2023-03-19T07:13:13.859 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2023-03-19T07:13:13.888 INFO:teuthology.lock.ops:Node 'smithi122.front.sepia.ceph.com' reimaging is complete 2023-03-19T07:13:17.320 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2023-03-19T07:13:17.320 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2023-03-19T07:13:17.320 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-03-19T07:13:17.354 INFO:teuthology.lock.ops:Node 'smithi150.front.sepia.ceph.com' reimaging is complete 2023-03-19T07:13:18.205 INFO:teuthology.lock.ops:Start node 'smithi184.front.sepia.ceph.com' reimaging 2023-03-19T07:13:18.205 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: reset os type and version on server 2023-03-19T07:13:18.205 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2023-03-19T07:13:18.229 INFO:teuthology.lock.ops:Node 'smithi184.front.sepia.ceph.com' reimaging is complete 2023-03-19T07:13:19.432 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of rhel 8.6 2023-03-19T07:13:21.884 INFO:teuthology.provision.fog.smithi112:Scheduling deploy of rhel 8.6 2023-03-19T07:13:23.272 INFO:teuthology.provision.fog.smithi122:Scheduling deploy of rhel 8.6 2023-03-19T07:13:25.858 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of rhel 8.6 2023-03-19T07:13:27.640 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of rhel 8.6 2023-03-19T07:13:29.138 INFO:teuthology.orchestra.console:Power off smithi122 2023-03-19T07:13:29.719 INFO:teuthology.orchestra.console:Power off smithi072 2023-03-19T07:13:31.446 INFO:teuthology.orchestra.console:Power off smithi112 2023-03-19T07:13:34.640 INFO:teuthology.orchestra.console:Power off smithi150 2023-03-19T07:13:36.354 INFO:teuthology.orchestra.console:Power off smithi184 2023-03-19T07:13:39.168 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2023-03-19T07:13:40.499 INFO:teuthology.orchestra.console:Power on smithi122 2023-03-19T07:13:43.271 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2023-03-19T07:13:44.492 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2023-03-19T07:13:44.592 INFO:teuthology.orchestra.console:Power on smithi072 2023-03-19T07:13:46.431 INFO:teuthology.orchestra.console:Power on smithi112 2023-03-19T07:13:48.755 INFO:teuthology.orchestra.console:Power on for smithi122 completed 2023-03-19T07:13:50.063 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-03-19T07:13:50.063 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2023-03-19T07:13:50.163 INFO:teuthology.provision.fog.smithi122:Waiting for deploy to finish 2023-03-19T07:13:51.751 INFO:teuthology.orchestra.console:Power on smithi150 2023-03-19T07:13:51.759 INFO:teuthology.orchestra.console:Power on smithi184 2023-03-19T07:13:54.424 INFO:teuthology.orchestra.console:Power on for smithi072 completed 2023-03-19T07:13:56.483 INFO:teuthology.orchestra.console:Power on for smithi112 completed 2023-03-19T07:13:56.583 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish 2023-03-19T07:13:58.286 INFO:teuthology.provision.fog.smithi112:Waiting for deploy to finish 2023-03-19T07:13:58.499 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2023-03-19T07:13:59.333 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2023-03-19T07:14:02.616 INFO:teuthology.orchestra.console:Power on for smithi184 completed 2023-03-19T07:14:03.302 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish 2023-03-19T07:14:19.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:14:31.813 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:14:43.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:14:53.993 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:15:04.869 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:15:30.693 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:15:40.104 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:15:46.978 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:15:49.801 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:15:51.983 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:15:54.652 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:15:56.910 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:00.585 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:02.871 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:05.238 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:07.612 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:11.958 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:13.277 WARNING:teuthology.provision.fog:'connect to smithi150.front.sepia.ceph.com' reached maximum tries (10) after waiting for 10 seconds 2023-03-19T07:16:19.347 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:22.459 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner 2023-03-19T07:16:23.100 ERROR:paramiko.transport:Traceback (most recent call last): 2023-03-19T07:16:23.100 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner 2023-03-19T07:16:23.100 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2023-03-19T07:16:23.100 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline 2023-03-19T07:16:23.100 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2023-03-19T07:16:23.100 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout 2023-03-19T07:16:23.100 ERROR:paramiko.transport: raise EOFError() 2023-03-19T07:16:23.100 ERROR:paramiko.transport:EOFError 2023-03-19T07:16:23.100 ERROR:paramiko.transport: 2023-03-19T07:16:25.736 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2023-03-19T07:16:25.736 ERROR:paramiko.transport: 2023-03-19T07:16:25.736 ERROR:paramiko.transport:Traceback (most recent call last): 2023-03-19T07:16:25.736 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run 2023-03-19T07:16:25.736 ERROR:paramiko.transport: self._check_banner() 2023-03-19T07:16:25.736 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner 2023-03-19T07:16:25.736 ERROR:paramiko.transport: raise SSHException( 2023-03-19T07:16:25.736 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2023-03-19T07:16:25.736 ERROR:paramiko.transport: 2023-03-19T07:16:25.736 WARNING:teuthology.provision.fog:Error reading SSH protocol banner 2023-03-19T07:16:30.537 ERROR:teuthology.orchestra.connection:Error authenticating with smithi112.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:33.784 ERROR:teuthology.orchestra.connection:Error authenticating with smithi150.front.sepia.ceph.com: Authentication failed. 2023-03-19T07:16:36.710 ERROR:paramiko.transport:Exception (client): Error reading SSH protocol banner 2023-03-19T07:16:38.540 ERROR:paramiko.transport:Traceback (most recent call last): 2023-03-19T07:16:38.540 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2270, in _check_banner 2023-03-19T07:16:38.540 ERROR:paramiko.transport: buf = self.packetizer.readline(timeout) 2023-03-19T07:16:38.540 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 380, in readline 2023-03-19T07:16:38.540 ERROR:paramiko.transport: buf += self._read_timeout(timeout) 2023-03-19T07:16:38.540 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 609, in _read_timeout 2023-03-19T07:16:38.540 ERROR:paramiko.transport: raise EOFError() 2023-03-19T07:16:38.541 ERROR:paramiko.transport:EOFError 2023-03-19T07:16:38.541 ERROR:paramiko.transport: 2023-03-19T07:16:38.541 ERROR:paramiko.transport:During handling of the above exception, another exception occurred: 2023-03-19T07:16:38.541 ERROR:paramiko.transport: 2023-03-19T07:16:38.541 ERROR:paramiko.transport:Traceback (most recent call last): 2023-03-19T07:16:38.541 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2093, in run 2023-03-19T07:16:38.541 ERROR:paramiko.transport: self._check_banner() 2023-03-19T07:16:38.541 ERROR:paramiko.transport: File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2274, in _check_banner 2023-03-19T07:16:38.541 ERROR:paramiko.transport: raise SSHException( 2023-03-19T07:16:38.541 ERROR:paramiko.transport:paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2023-03-19T07:16:38.541 ERROR:paramiko.transport: 2023-03-19T07:16:38.541 WARNING:teuthology.provision.fog:Error reading SSH protocol banner 2023-03-19T07:16:42.377 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-03-19T07:16:46.217 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-19T07:16:54.021 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-03-19T07:16:58.629 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-19T07:17:03.369 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-03-19T07:17:06.721 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-03-19T07:17:09.801 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-19T07:17:15.781 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-03-19T07:17:16.839 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-03-19T07:17:18.981 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-03-19T07:17:20.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.122 2023-03-19T07:17:25.897 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-03-19T07:17:26.835 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.112 2023-03-19T07:17:29.893 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-03-19T07:17:35.909 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.184 2023-03-19T07:17:39.941 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-03-19T07:17:44.953 WARNING:teuthology.provision.fog:timed out 2023-03-19T07:17:51.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-03-19T07:18:11.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:18:18.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-03-19T07:18:21.925 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.150 2023-03-19T07:18:26.839 WARNING:teuthology.provision.fog:timed out 2023-03-19T07:18:28.998 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-03-19T07:18:29.549 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-19T07:18:31.627 INFO:teuthology.provision.fog.smithi150:Node is ready 2023-03-19T07:18:32.524 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com 2023-03-19T07:18:33.674 WARNING:teuthology.provision.fog:timed out 2023-03-19T07:18:36.292 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150 2023-03-19T07:18:38.396 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-19T07:18:39.236 INFO:teuthology.provision.fog.smithi150:Deploy complete! 2023-03-19T07:18:39.711 INFO:teuthology.provision.fog.smithi122:Node is ready 2023-03-19T07:18:40.193 INFO:teuthology.orchestra.run.smithi122.stdout:smithi122 2023-03-19T07:18:41.775 INFO:teuthology.orchestra.run.smithi122.stdout:172.21.15.122 smithi122.front.sepia.ceph.com smithi122 2023-03-19T07:18:43.182 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-19T07:18:44.375 WARNING:teuthology.provision.fog:timed out 2023-03-19T07:18:44.497 INFO:teuthology.provision.fog.smithi072:Node is ready 2023-03-19T07:18:47.050 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-19T07:18:47.070 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072.front.sepia.ceph.com 2023-03-19T07:18:47.770 INFO:teuthology.provision.fog.smithi112:Node is ready 2023-03-19T07:18:49.916 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072 2023-03-19T07:18:51.006 INFO:teuthology.orchestra.run.smithi112.stdout:smithi112 2023-03-19T07:18:52.149 INFO:teuthology.orchestra.run.smithi112.stdout:172.21.15.112 smithi112.front.sepia.ceph.com smithi112 2023-03-19T07:18:53.148 INFO:teuthology.provision.fog.smithi122:Deploy complete! 2023-03-19T07:18:55.342 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-19T07:18:55.513 INFO:teuthology.provision.fog.smithi072:Deploy complete! 2023-03-19T07:18:59.923 INFO:teuthology.provision.fog.smithi184:Node is ready 2023-03-19T07:18:59.942 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184.front.sepia.ceph.com 2023-03-19T07:19:02.118 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.15.184 smithi184.front.sepia.ceph.com smithi184 2023-03-19T07:19:03.869 INFO:teuthology.provision.fog.smithi112:Deploy complete! 2023-03-19T07:19:05.659 INFO:teuthology.provision.fog.smithi184:Deploy complete! 2023-03-19T07:19:07.033 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com 2023-03-19T07:19:07.400 INFO:teuthology.lock.ops:Checking smithi112.front.sepia.ceph.com 2023-03-19T07:19:07.400 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-19T07:19:07.425 INFO:teuthology.lock.ops:Checking smithi122.front.sepia.ceph.com 2023-03-19T07:19:08.298 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-19T07:19:08.325 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com 2023-03-19T07:19:10.398 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-19T07:19:10.424 INFO:teuthology.lock.ops:Checking smithi184.front.sepia.ceph.com 2023-03-19T07:19:11.918 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-19T07:19:12.003 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server 2023-03-19T07:19:14.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T07:19:14.635 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2023-03-19T07:19:16.665 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="Red Hat Enterprise Linux" 2023-03-19T07:19:19.022 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8.6 (Ootpa)" 2023-03-19T07:19:19.634 INFO:teuthology.orchestra.run.smithi072.stdout:ID="rhel" 2023-03-19T07:19:19.635 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="fedora" 2023-03-19T07:19:19.635 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8.6" 2023-03-19T07:19:19.635 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2023-03-19T07:19:19.635 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-19T07:19:19.635 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2023-03-19T07:19:19.635 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-19T07:19:19.635 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://www.redhat.com/" 2023-03-19T07:19:20.046 INFO:teuthology.orchestra.run.smithi072.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-19T07:19:20.046 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-19T07:19:20.047 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-03-19T07:19:20.047 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-19T07:19:20.047 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-19T07:19:20.047 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-19T07:19:20.047 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-19T07:19:20.048 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-03-19T07:19:21.706 INFO:teuthology.lock.ops:Updating [smithi112.front.sepia.ceph.com]: set os type and version on server 2023-03-19T07:19:22.685 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T07:19:22.938 INFO:teuthology.orchestra.run.smithi112.stdout:x86_64 2023-03-19T07:19:24.512 INFO:teuthology.orchestra.run.smithi112.stdout:NAME="Red Hat Enterprise Linux" 2023-03-19T07:19:25.070 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION="8.6 (Ootpa)" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:ID="rhel" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:ID_LIKE="fedora" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:VERSION_ID="8.6" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:PLATFORM_ID="platform:el8" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:ANSI_COLOR="0;31" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-19T07:19:25.071 INFO:teuthology.orchestra.run.smithi112.stdout:HOME_URL="https://www.redhat.com/" 2023-03-19T07:19:25.072 INFO:teuthology.orchestra.run.smithi112.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-19T07:19:25.072 INFO:teuthology.orchestra.run.smithi112.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-19T07:19:25.072 INFO:teuthology.orchestra.run.smithi112.stdout: 2023-03-19T07:19:25.072 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-19T07:19:25.072 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-19T07:19:25.072 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-19T07:19:25.072 INFO:teuthology.orchestra.run.smithi112.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-19T07:19:25.073 INFO:teuthology.lock.ops:Updating smithi112.front.sepia.ceph.com on lock server 2023-03-19T07:19:25.097 INFO:teuthology.lock.ops:Updating [smithi122.front.sepia.ceph.com]: set os type and version on server 2023-03-19T07:19:25.751 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T07:19:25.967 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2023-03-19T07:19:26.365 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Red Hat Enterprise Linux" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8.6 (Ootpa)" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:ID="rhel" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="fedora" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8.6" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2023-03-19T07:19:26.940 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.redhat.com/" 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-19T07:19:26.941 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-19T07:19:26.942 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2023-03-19T07:19:26.964 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server 2023-03-19T07:19:27.279 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T07:19:27.534 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2023-03-19T07:19:27.972 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Red Hat Enterprise Linux" 2023-03-19T07:19:28.880 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8.6 (Ootpa)" 2023-03-19T07:19:29.122 INFO:teuthology.orchestra.run.smithi150.stdout:ID="rhel" 2023-03-19T07:19:29.122 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="fedora" 2023-03-19T07:19:29.122 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8.6" 2023-03-19T07:19:29.123 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2023-03-19T07:19:29.123 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-19T07:19:29.123 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2023-03-19T07:19:29.123 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-19T07:19:29.123 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.redhat.com/" 2023-03-19T07:19:29.123 INFO:teuthology.orchestra.run.smithi150.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-19T07:19:30.482 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-19T07:19:30.482 INFO:teuthology.orchestra.run.smithi150.stdout: 2023-03-19T07:19:30.482 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-19T07:19:30.482 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-19T07:19:30.482 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-19T07:19:30.483 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-19T07:19:30.483 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2023-03-19T07:19:30.505 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: set os type and version on server 2023-03-19T07:19:32.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T07:19:32.925 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64 2023-03-19T07:19:33.924 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Red Hat Enterprise Linux" 2023-03-19T07:19:35.164 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="8.6 (Ootpa)" 2023-03-19T07:19:35.164 INFO:teuthology.orchestra.run.smithi184.stdout:ID="rhel" 2023-03-19T07:19:35.164 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE="fedora" 2023-03-19T07:19:35.164 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="8.6" 2023-03-19T07:19:35.164 INFO:teuthology.orchestra.run.smithi184.stdout:PLATFORM_ID="platform:el8" 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout:ANSI_COLOR="0;31" 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.redhat.com/" 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout: 2023-03-19T07:19:35.165 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-19T07:19:35.166 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-19T07:19:35.166 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-19T07:19:35.166 INFO:teuthology.orchestra.run.smithi184.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-19T07:19:35.166 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server 2023-03-19T07:19:38.370 INFO:teuthology.dispatcher.supervisor:Running job 7212277 2023-03-19T07:19:41.996 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi/7212277 --name yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi --description rados/cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli_mon} -- /home/teuthworker/archive/yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi/7212277/orig.config.yaml 2023-03-19T07:19:42.000 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-03-17_23:38:21-rados-reef-distro-default-smithi/7212277 2023-03-19T07:19:42.001 INFO:teuthology.dispatcher.supervisor:Job PID: 26886 2023-03-19T07:19:42.001 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-03-19T19:20:05.391 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:20:06.180 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:20:08.757 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:20:09.775 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:20:09.775 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:20:09.775 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:20:09.775 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:20:09.775 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:20:09.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:20:09.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:20:12.965 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:20:14.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:20:17.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:20:19.943 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:20:21.279 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-03-19T19:20:21.430 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-03-19T19:20:22.739 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-03-19T19:20:23.759 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-03-19T19:20:24.355 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:20:25.842 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:20:27.675 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:20:28.355 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:20:28.996 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:20:30.269 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:22:32.002 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:22:32.823 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:22:33.615 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:22:34.853 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:22:36.067 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:22:36.678 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:22:36.678 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:22:37.663 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:22:37.663 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:22:37.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:22:38.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:22:39.704 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:22:40.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:22:42.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:22:43.484 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:22:45.853 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:22:47.105 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:22:49.136 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:22:49.137 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:22:49.138 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:22:49.138 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:22:49.138 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:22:49.139 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:22:51.114 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:22:53.271 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:24:56.412 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:24:59.029 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:25:00.469 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:25:02.919 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:25:04.955 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:25:04.955 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:25:04.955 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:25:04.955 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:25:04.955 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:25:04.955 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:25:06.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:25:07.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:25:09.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:25:12.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:25:13.539 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:25:15.385 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:25:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:25:18.678 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:25:18.679 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:25:18.679 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:25:18.679 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:25:18.679 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:25:18.680 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:25:20.377 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:25:21.531 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:27:22.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:27:27.135 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:27:29.512 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:27:31.581 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:27:34.062 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:27:34.062 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:27:34.062 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:27:34.062 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:27:34.062 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:27:34.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:27:35.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:27:38.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:27:40.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:27:41.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:27:43.620 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:27:48.203 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:27:50.503 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:27:51.352 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:27:51.353 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:27:51.354 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:27:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:27:51.354 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:27:51.747 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:27:56.274 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:28:00.301 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:30:02.334 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:30:06.700 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:30:08.683 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:30:09.428 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:30:10.928 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:30:10.928 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:30:10.928 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:30:10.928 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:30:10.928 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:30:10.928 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:30:11.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:30:12.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:30:14.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:30:15.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:30:17.729 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:30:19.638 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:30:21.413 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:30:21.720 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:30:21.721 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:30:23.414 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:30:24.039 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:30:24.039 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:30:24.040 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:30:25.685 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:30:27.251 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:32:29.220 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:32:33.100 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:32:34.884 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:32:36.813 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:32:37.770 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:32:37.770 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:32:38.182 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:32:38.182 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:32:38.182 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:32:38.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:32:38.364 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:32:39.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:32:40.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:32:40.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:32:42.978 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:32:47.105 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:32:48.755 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:32:50.918 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:32:50.919 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:32:50.920 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:32:52.128 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:32:52.128 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:32:52.129 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:32:52.167 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:32:53.319 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:34:54.902 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:34:55.609 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:34:56.720 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:34:58.250 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:34:59.073 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:34:59.073 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:34:59.073 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:34:59.073 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:34:59.073 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:34:59.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:34:59.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:35:02.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:35:03.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:35:04.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:35:07.943 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:35:09.603 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:35:10.384 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:35:12.110 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:35:12.111 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:35:12.700 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:35:12.700 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:35:12.700 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:35:13.600 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:35:16.358 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:35:17.988 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:37:21.589 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:37:26.827 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:37:27.772 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:37:28.275 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:37:30.020 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:37:30.020 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:37:30.020 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:37:30.020 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:37:30.020 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:37:30.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:37:30.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:37:31.400 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:37:32.440 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:37:33.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:37:34.984 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:37:39.413 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:37:41.876 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:37:43.108 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:37:43.110 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:37:43.110 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:37:43.110 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:37:43.624 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:37:43.625 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:37:43.663 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:37:44.474 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:39:46.690 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:39:47.487 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:39:48.318 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:39:51.609 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:39:52.217 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:39:52.218 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:39:52.218 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:39:52.218 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:39:52.218 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:39:52.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:39:52.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:39:53.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:39:54.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:39:56.168 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:39:59.599 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:40:02.042 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:40:02.961 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:40:04.386 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:40:04.387 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:40:04.388 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:40:04.388 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:40:04.388 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:40:05.063 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi072 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:40:06.320 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:40:07.983 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:42:10.003 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:42:17.213 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:42:18.908 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:42:19.571 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:42:21.046 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:42:21.662 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:42:21.662 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:42:21.662 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:42:21.662 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:42:22.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:42:22.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:42:23.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:42:24.268 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:42:26.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:42:30.366 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:42:31.845 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:42:32.759 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:42:33.947 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:42:33.948 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:42:33.948 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:42:40.769 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2023-03-19T19:42:42.754 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.8.log: file size changed while zipping 2023-03-19T19:42:42.754 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.10.log: file size changed while zipping 2023-03-19T19:42:43.240 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.13.log: file size changed while zipping 2023-03-19T19:42:49.630 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.11.log: file size changed while zipping 2023-03-19T19:42:51.967 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.12.log: file size changed while zipping 2023-03-19T19:42:51.967 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2023-03-19T19:42:53.725 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-mon.smithi150.log: file size changed while zipping 2023-03-19T19:42:54.632 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2023-03-19T19:42:58.310 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-osd.14.log: file size changed while zipping 2023-03-19T19:43:02.520 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-mon.smithi184.log: file size changed while zipping 2023-03-19T19:43:04.295 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:43:04.338 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:43:06.575 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:45:08.194 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:45:11.125 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:45:12.443 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:45:13.391 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:45:14.161 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:45:15.338 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:45:16.967 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:45:16.967 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:45:16.967 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:45:16.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:45:17.762 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:45:19.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:45:21.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:45:22.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:45:24.234 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:45:27.716 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:45:32.590 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:45:34.887 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:45:34.888 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:45:34.888 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:45:34.888 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:45:34.889 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:45:34.889 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:45:34.927 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:45:36.514 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:47:39.008 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:47:40.095 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:47:41.229 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:47:43.015 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:47:45.975 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:47:45.975 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:47:45.975 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:47:45.975 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:47:45.975 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:47:45.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:47:46.155 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:47:48.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:47:50.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:47:53.211 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:47:56.593 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:47:59.016 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:48:00.731 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:48:02.906 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:48:02.908 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:48:02.909 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:48:02.909 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:48:02.911 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:48:05.127 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:48:05.879 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:48:07.330 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:50:08.658 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:50:09.701 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:50:11.083 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:50:12.907 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:50:14.068 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:50:15.341 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:50:15.341 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:50:15.341 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:50:15.341 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:50:15.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:50:15.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:50:17.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:50:18.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:50:19.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:50:20.824 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:50:23.180 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:50:24.207 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:50:26.037 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:50:28.031 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:50:30.030 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:50:30.030 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:50:30.030 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:50:30.031 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:50:30.765 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:50:32.779 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:52:33.476 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:52:34.248 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:52:36.632 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:52:37.324 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:52:37.897 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:52:38.903 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:52:38.903 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:52:38.904 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:52:38.904 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:52:38.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:52:39.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:52:40.956 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:52:43.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:52:45.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:52:51.236 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:52:54.194 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:52:57.406 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:53:01.662 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:53:01.663 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:53:01.664 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:53:01.664 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:53:01.665 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:53:01.665 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:53:01.705 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:53:03.754 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:55:06.102 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:55:06.745 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:55:07.634 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:55:07.987 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:55:08.967 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:55:08.967 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:55:08.967 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:55:08.967 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:55:08.967 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:55:08.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:55:09.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:55:10.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:55:11.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:55:11.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:55:12.413 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:55:14.528 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:55:15.198 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:55:15.670 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:55:16.401 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:55:16.401 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:55:16.402 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:55:18.040 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:55:18.040 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:55:18.077 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:55:19.274 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:57:21.626 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-19T19:57:24.336 INFO:teuthology.kill:Killing Pids: {26886} 2023-03-19T19:57:27.090 ERROR:teuthology.dispatcher.supervisor:Failed to kill job Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 288, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:57:27.111 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-03-19T19:57:28.336 INFO:teuthology.task.internal:roles: ubuntu@smithi112.front.sepia.ceph.com - ['host.b', 'osd.3', 'osd.4', 'osd.5', 'mon.b', 'mgr.b', 'client.1'] 2023-03-19T19:57:28.337 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['host.c', 'osd.6', 'osd.7', 'osd.8', 'mon.c', 'mgr.c', 'client.2'] 2023-03-19T19:57:28.337 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.d', 'osd.9', 'osd.10', 'osd.11', 'mon.d', 'mgr.d', 'client.3'] 2023-03-19T19:57:28.337 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['host.e', 'osd.12', 'osd.13', 'osd.14', 'mon.e', 'mgr.e', 'client.4'] 2023-03-19T19:57:28.337 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:57:28.337 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-03-19T19:57:28.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi112.front.sepia.ceph.com' 2023-03-19T19:57:29.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-19T19:57:30.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2023-03-19T19:57:31.325 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com' 2023-03-19T19:57:33.131 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:57:34.093 INFO:teuthology.misc:Compressing logs... 2023-03-19T19:57:35.032 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:57:36.894 INFO:teuthology.orchestra.run.smithi112.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:57:36.896 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:57:41.428 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:57:41.428 INFO:teuthology.orchestra.run.smithi122.stderr:gzip: /var/log/ceph/4d697418-c62a-11ed-9afc-001a4aab830c/ceph-volume.log.gz already exists; not overwritten 2023-03-19T19:57:41.429 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /var/log/ceph/cephadm.log.gz already exists; not overwritten 2023-03-19T19:57:42.439 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 295, in run_with_watchdog transfer_archives(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 358, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/misc.py", line 1367, in compress_logs run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi112 with status 123: 'sudo find /var/log/ceph -name *.log -print0 | sudo xargs -0 --no-run-if-empty -- gzip --' 2023-03-19T19:57:43.363 INFO:teuthology.kill:No teuthology processes running 2023-03-19T19:57:44.493 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog kill_job(job_info['name'], job_info['job_id'], File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/kill.py", line 90, in kill_job teuthology.exporter.JobResults().record( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/teuthology/exporter.py", line 175, in __init__ self.metric = Counter( File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 143, in __init__ registry.register(self) File "/home/teuthworker/src/git.ceph.com_teuthology_7f7f7c33a372789d0252bbbcd5798300b0f044d0/virtualenv/lib/python3.8/site-packages/prometheus_client/registry.py", line 43, in register raise ValueError( ValueError: Duplicated timeseries in CollectorRegistry: {'teuthology_job_results_total', 'teuthology_job_results_created', 'teuthology_job_results'} 2023-03-19T19:59:44.633 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-03-19T19:59:45.115 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2023-03-19T19:59:45.944 INFO:teuthology.nuke:Checking targets against current locks 2023-03-19T19:59:46.003 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-19T19:59:47.220 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-19T19:59:49.020 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-19T19:59:50.037 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-19T19:59:51.996 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-19T19:59:53.201 INFO:teuthology.orchestra.console:Power off smithi072 2023-03-19T19:59:55.028 INFO:teuthology.orchestra.console:Power off smithi122 2023-03-19T19:59:56.845 INFO:teuthology.orchestra.console:Power off smithi112 2023-03-19T19:59:58.644 INFO:teuthology.orchestra.console:Power off smithi150 2023-03-19T20:00:01.058 INFO:teuthology.orchestra.console:Power off smithi184 2023-03-19T20:00:06.401 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2023-03-19T20:00:08.613 INFO:teuthology.orchestra.console:Power off for smithi122 completed 2023-03-19T20:00:08.770 INFO:teuthology.lock.ops:unlocked: smithi122.front.sepia.ceph.com 2023-03-19T20:00:10.559 INFO:teuthology.lock.ops:unlocked: smithi072.front.sepia.ceph.com 2023-03-19T20:00:11.584 INFO:teuthology.orchestra.console:Power off for smithi112 completed 2023-03-19T20:00:12.354 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2023-03-19T20:00:13.727 INFO:teuthology.lock.ops:unlocked: smithi112.front.sepia.ceph.com 2023-03-19T20:00:15.376 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2023-03-19T20:00:17.068 INFO:teuthology.orchestra.console:Power off for smithi184 completed 2023-03-19T20:00:19.061 INFO:teuthology.lock.ops:unlocked: smithi184.front.sepia.ceph.com