2026-02-12T15:49:03.782 INFO:root:teuthology version: 1.2.4.dev6+g96afe742a
2026-02-12T15:49:03.782 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T15:49:03.855 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T15:49:04.103 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x754c04f0c350>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x754c04e64c20>, signals=[15])
2026-02-12T15:49:04.103 INFO:teuthology.lock.ops:Start node 'trial020.front.sepia.ceph.com' reimaging
2026-02-12T15:49:04.103 INFO:teuthology.lock.ops:Updating [trial020.front.sepia.ceph.com]: reset os type and version on server
2026-02-12T15:49:04.103 INFO:teuthology.lock.ops:Updating trial020.front.sepia.ceph.com on lock server
2026-02-12T15:49:04.158 INFO:teuthology.lock.ops:Start node 'trial030.front.sepia.ceph.com' reimaging
2026-02-12T15:49:04.158 INFO:teuthology.lock.ops:Updating [trial030.front.sepia.ceph.com]: reset os type and version on server
2026-02-12T15:49:04.158 INFO:teuthology.lock.ops:Updating trial030.front.sepia.ceph.com on lock server
2026-02-12T15:49:04.216 INFO:teuthology.lock.ops:Start node 'trial094.front.sepia.ceph.com' reimaging
2026-02-12T15:49:04.216 INFO:teuthology.lock.ops:Updating [trial094.front.sepia.ceph.com]: reset os type and version on server
2026-02-12T15:49:04.216 INFO:teuthology.lock.ops:Updating trial094.front.sepia.ceph.com on lock server
2026-02-12T15:49:04.449 DEBUG:teuthology.provision.fog.trial030:Requesting image trial_centos_9.stream (ID 3)
2026-02-12T15:49:04.449 DEBUG:teuthology.provision.fog.trial020:Requesting image trial_centos_9.stream (ID 3)
2026-02-12T15:49:04.503 INFO:teuthology.provision.fog.trial030:Scheduling deploy of centos 9.stream
2026-02-12T15:49:04.505 DEBUG:teuthology.provision.fog.trial094:Requesting image trial_centos_9.stream (ID 3)
2026-02-12T15:49:04.505 INFO:teuthology.provision.fog.trial020:Scheduling deploy of centos 9.stream
2026-02-12T15:49:04.557 INFO:teuthology.provision.fog.trial094:Scheduling deploy of centos 9.stream
2026-02-12T15:49:05.563 INFO:teuthology.orchestra.console.trial030:Power off
2026-02-12T15:49:05.563 DEBUG:teuthology.orchestra.console.trial030:pexpect command: ipmitool -H trial030.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T15:49:05.575 INFO:teuthology.orchestra.console.trial094:Power off
2026-02-12T15:49:05.575 DEBUG:teuthology.orchestra.console.trial094:pexpect command: ipmitool -H trial094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T15:49:05.584 INFO:teuthology.orchestra.console.trial020:Power off
2026-02-12T15:49:05.585 DEBUG:teuthology.orchestra.console.trial020:pexpect command: ipmitool -H trial020.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T15:49:05.602 DEBUG:teuthology.orchestra.console.trial030:power off output: Chassis Power Control: Down/Off
2026-02-12T15:49:05.613 DEBUG:teuthology.orchestra.console.trial094:power off output: Chassis Power Control: Down/Off
2026-02-12T15:49:05.622 DEBUG:teuthology.orchestra.console.trial020:power off output: Chassis Power Control: Down/Off
2026-02-12T15:49:09.606 DEBUG:teuthology.orchestra.console.trial030:pexpect command: ipmitool -H trial030.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T15:49:09.614 DEBUG:teuthology.orchestra.console.trial094:pexpect command: ipmitool -H trial094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T15:49:09.622 DEBUG:teuthology.orchestra.console.trial020:pexpect command: ipmitool -H trial020.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T15:49:09.713 DEBUG:teuthology.orchestra.console.trial030:check power output: Chassis Power is off
2026-02-12T15:49:09.714 INFO:teuthology.orchestra.console.trial030:Power off completed
2026-02-12T15:49:09.722 DEBUG:teuthology.orchestra.console.trial094:check power output: Chassis Power is off
2026-02-12T15:49:09.722 INFO:teuthology.orchestra.console.trial094:Power off completed
2026-02-12T15:49:09.730 DEBUG:teuthology.orchestra.console.trial020:check power output: Chassis Power is off
2026-02-12T15:49:09.730 INFO:teuthology.orchestra.console.trial020:Power off completed
2026-02-12T15:49:09.815 INFO:teuthology.orchestra.console.trial030:Power on
2026-02-12T15:49:09.815 DEBUG:teuthology.orchestra.console.trial030:pexpect command: ipmitool -H trial030.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-12T15:49:09.824 INFO:teuthology.orchestra.console.trial094:Power on
2026-02-12T15:49:09.824 DEBUG:teuthology.orchestra.console.trial094:pexpect command: ipmitool -H trial094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-12T15:49:09.832 INFO:teuthology.orchestra.console.trial020:Power on
2026-02-12T15:49:09.832 DEBUG:teuthology.orchestra.console.trial020:pexpect command: ipmitool -H trial020.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-12T15:49:09.855 DEBUG:teuthology.orchestra.console.trial030:power on output: Chassis Power Control: Up/On
2026-02-12T15:49:09.864 DEBUG:teuthology.orchestra.console.trial094:power on output: Chassis Power Control: Up/On
2026-02-12T15:49:09.870 DEBUG:teuthology.orchestra.console.trial020:power on output: Chassis Power Control: Up/On
2026-02-12T15:49:13.860 DEBUG:teuthology.orchestra.console.trial030:pexpect command: ipmitool -H trial030.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T15:49:13.868 DEBUG:teuthology.orchestra.console.trial094:pexpect command: ipmitool -H trial094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T15:49:13.876 DEBUG:teuthology.orchestra.console.trial020:pexpect command: ipmitool -H trial020.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T15:49:13.971 DEBUG:teuthology.orchestra.console.trial030:check power output: Chassis Power is on
2026-02-12T15:49:13.972 INFO:teuthology.orchestra.console.trial030:Power on completed
2026-02-12T15:49:13.978 DEBUG:teuthology.orchestra.console.trial094:check power output: Chassis Power is on
2026-02-12T15:49:13.978 INFO:teuthology.orchestra.console.trial094:Power on completed
2026-02-12T15:49:13.985 DEBUG:teuthology.orchestra.console.trial020:check power output: Chassis Power is on
2026-02-12T15:49:13.986 INFO:teuthology.orchestra.console.trial020:Power on completed
2026-02-12T15:49:14.073 INFO:teuthology.provision.fog.trial030:Waiting for deploy to finish
2026-02-12T15:49:14.079 INFO:teuthology.provision.fog.trial094:Waiting for deploy to finish
2026-02-12T15:49:14.087 INFO:teuthology.provision.fog.trial020:Waiting for deploy to finish
2026-02-12T15:51:01.482 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:01.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:04.541 WARNING:teuthology.provision.fog.trial030:[Errno None] Unable to connect to port 22 on 10.20.193.30
2026-02-12T15:51:10.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:13.629 WARNING:teuthology.provision.fog.trial030:[Errno None] Unable to connect to port 22 on 10.20.193.30
2026-02-12T15:51:16.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:19.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:19.773 WARNING:teuthology.provision.fog.trial094:[Errno None] Unable to connect to port 22 on 10.20.193.94
2026-02-12T15:51:22.717 WARNING:teuthology.provision.fog.trial030:[Errno None] Unable to connect to port 22 on 10.20.193.30
2026-02-12T15:51:25.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:28.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:28.861 WARNING:teuthology.provision.fog.trial094:[Errno None] Unable to connect to port 22 on 10.20.193.94
2026-02-12T15:51:31.805 WARNING:teuthology.provision.fog.trial030:[Errno None] Unable to connect to port 22 on 10.20.193.30
2026-02-12T15:51:34.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:37.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:51:37.949 WARNING:teuthology.provision.fog.trial094:[Errno None] Unable to connect to port 22 on 10.20.193.94
2026-02-12T15:51:43.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:52:01.507 WARNING:teuthology.provision.fog.trial020:timed out
2026-02-12T15:52:07.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:52:07.652 ERROR:teuthology.orchestra.connection:Error authenticating with trial020.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-12T15:52:11.741 ERROR:teuthology.orchestra.connection:Error authenticating with trial020.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-12T15:52:13.831 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-12T15:52:13.831 DEBUG:teuthology.orchestra.run.trial030:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-12T15:52:14.072 INFO:teuthology.provision.fog.trial030:Node is ready
2026-02-12T15:52:14.073 DEBUG:teuthology.orchestra.run.trial030:> hostname
2026-02-12T15:52:14.130 INFO:teuthology.orchestra.run.trial030.stdout:trial030.front.sepia.ceph.com
2026-02-12T15:52:14.130 DEBUG:teuthology.orchestra.run.trial030:> grep trial030.front.sepia.ceph.com /etc/hosts
2026-02-12T15:52:14.186 INFO:teuthology.orchestra.run.trial030.stdout:10.20.193.30 trial030.front.sepia.ceph.com trial030
2026-02-12T15:52:14.186 DEBUG:teuthology.orchestra.run.trial030:> sudo hostname trial030
2026-02-12T15:52:14.255 DEBUG:teuthology.orchestra.run.trial030:> sudo sed -i -e 's/trial030.front.sepia.ceph.com/trial030/g' /etc/hosts
2026-02-12T15:52:14.319 DEBUG:teuthology.orchestra.run.trial030:> sudo sed -i -e 's/10.20.193.30/10.20.193.30/g' /etc/hosts
2026-02-12T15:52:14.384 DEBUG:teuthology.orchestra.run.trial030:> sudo sed -i -e 's/trial030.front.sepia.ceph.com/trial030/g' /etc/hostname
2026-02-12T15:52:14.450 DEBUG:teuthology.orchestra.run.trial030:> sudo hostname trial030
2026-02-12T15:52:14.516 DEBUG:teuthology.orchestra.run.trial030:> cat /etc/os-release
2026-02-12T15:52:14.572 INFO:teuthology.orchestra.run.trial030.stdout:NAME="CentOS Stream"
2026-02-12T15:52:14.572 INFO:teuthology.orchestra.run.trial030.stdout:VERSION="9"
2026-02-12T15:52:14.572 INFO:teuthology.orchestra.run.trial030.stdout:ID="centos"
2026-02-12T15:52:14.572 INFO:teuthology.orchestra.run.trial030.stdout:ID_LIKE="rhel fedora"
2026-02-12T15:52:14.572 INFO:teuthology.orchestra.run.trial030.stdout:VERSION_ID="9"
2026-02-12T15:52:14.572 INFO:teuthology.orchestra.run.trial030.stdout:PLATFORM_ID="platform:el9"
2026-02-12T15:52:14.572 INFO:teuthology.orchestra.run.trial030.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T15:52:14.573 INFO:teuthology.orchestra.run.trial030.stdout:ANSI_COLOR="0;31"
2026-02-12T15:52:14.573 INFO:teuthology.orchestra.run.trial030.stdout:LOGO="fedora-logo-icon"
2026-02-12T15:52:14.573 INFO:teuthology.orchestra.run.trial030.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T15:52:14.573 INFO:teuthology.orchestra.run.trial030.stdout:HOME_URL="https://centos.org/"
2026-02-12T15:52:14.573 INFO:teuthology.orchestra.run.trial030.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T15:52:14.573 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T15:52:14.573 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T15:52:14.574 INFO:teuthology.provision.fog.trial030:Deploy complete!
2026-02-12T15:52:14.575 DEBUG:teuthology.parallel:result is None
2026-02-12T15:52:18.857 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-12T15:52:18.857 DEBUG:teuthology.orchestra.run.trial020:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-12T15:52:19.020 INFO:teuthology.provision.fog.trial020:Node is ready
2026-02-12T15:52:19.020 DEBUG:teuthology.orchestra.run.trial020:> hostname
2026-02-12T15:52:19.076 INFO:teuthology.orchestra.run.trial020.stdout:trial020.front.sepia.ceph.com
2026-02-12T15:52:19.077 DEBUG:teuthology.orchestra.run.trial020:> grep trial020.front.sepia.ceph.com /etc/hosts
2026-02-12T15:52:19.133 INFO:teuthology.orchestra.run.trial020.stdout:10.20.193.20 trial020.front.sepia.ceph.com trial020
2026-02-12T15:52:19.134 DEBUG:teuthology.orchestra.run.trial020:> sudo hostname trial020
2026-02-12T15:52:19.203 DEBUG:teuthology.orchestra.run.trial020:> sudo sed -i -e 's/trial020.front.sepia.ceph.com/trial020/g' /etc/hosts
2026-02-12T15:52:19.268 DEBUG:teuthology.orchestra.run.trial020:> sudo sed -i -e 's/10.20.193.20/10.20.193.20/g' /etc/hosts
2026-02-12T15:52:19.332 DEBUG:teuthology.orchestra.run.trial020:> sudo sed -i -e 's/trial020.front.sepia.ceph.com/trial020/g' /etc/hostname
2026-02-12T15:52:19.397 DEBUG:teuthology.orchestra.run.trial020:> sudo hostname trial020
2026-02-12T15:52:19.462 DEBUG:teuthology.orchestra.run.trial020:> cat /etc/os-release
2026-02-12T15:52:19.518 INFO:teuthology.orchestra.run.trial020.stdout:NAME="CentOS Stream"
2026-02-12T15:52:19.518 INFO:teuthology.orchestra.run.trial020.stdout:VERSION="9"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:ID="centos"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:ID_LIKE="rhel fedora"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:VERSION_ID="9"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:PLATFORM_ID="platform:el9"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:ANSI_COLOR="0;31"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:LOGO="fedora-logo-icon"
2026-02-12T15:52:19.519 INFO:teuthology.orchestra.run.trial020.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T15:52:19.520 INFO:teuthology.orchestra.run.trial020.stdout:HOME_URL="https://centos.org/"
2026-02-12T15:52:19.520 INFO:teuthology.orchestra.run.trial020.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T15:52:19.520 INFO:teuthology.orchestra.run.trial020.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T15:52:19.520 INFO:teuthology.orchestra.run.trial020.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T15:52:19.520 INFO:teuthology.provision.fog.trial020:Deploy complete!
2026-02-12T15:52:19.521 DEBUG:teuthology.parallel:result is None
2026-02-12T15:52:20.020 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-12T15:52:20.020 DEBUG:teuthology.orchestra.run.trial094:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-12T15:52:20.195 INFO:teuthology.provision.fog.trial094:Node is ready
2026-02-12T15:52:20.196 DEBUG:teuthology.orchestra.run.trial094:> hostname
2026-02-12T15:52:20.252 INFO:teuthology.orchestra.run.trial094.stdout:trial094.front.sepia.ceph.com
2026-02-12T15:52:20.253 DEBUG:teuthology.orchestra.run.trial094:> grep trial094.front.sepia.ceph.com /etc/hosts
2026-02-12T15:52:20.309 INFO:teuthology.orchestra.run.trial094.stdout:10.20.193.94 trial094.front.sepia.ceph.com trial094
2026-02-12T15:52:20.309 DEBUG:teuthology.orchestra.run.trial094:> sudo hostname trial094
2026-02-12T15:52:20.378 DEBUG:teuthology.orchestra.run.trial094:> sudo sed -i -e 's/trial094.front.sepia.ceph.com/trial094/g' /etc/hosts
2026-02-12T15:52:20.443 DEBUG:teuthology.orchestra.run.trial094:> sudo sed -i -e 's/10.20.193.94/10.20.193.94/g' /etc/hosts
2026-02-12T15:52:20.508 DEBUG:teuthology.orchestra.run.trial094:> sudo sed -i -e 's/trial094.front.sepia.ceph.com/trial094/g' /etc/hostname
2026-02-12T15:52:20.573 DEBUG:teuthology.orchestra.run.trial094:> sudo hostname trial094
2026-02-12T15:52:20.636 DEBUG:teuthology.orchestra.run.trial094:> cat /etc/os-release
2026-02-12T15:52:20.689 ERROR:paramiko.transport:Unknown exception: The socket has already been closed by another greenlet
2026-02-12T15:52:20.692 ERROR:paramiko.transport:Traceback (most recent call last):
2026-02-12T15:52:20.692 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 2159, in run
2026-02-12T15:52:20.692 ERROR:paramiko.transport:    ptype, m = self.packetizer.read_message()
2026-02-12T15:52:20.692 ERROR:paramiko.transport:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-12T15:52:20.692 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/packet.py", line 463, in read_message
2026-02-12T15:52:20.692 ERROR:paramiko.transport:    header = self.read_all(self.__block_size_in, check_rekey=True)
2026-02-12T15:52:20.692 ERROR:paramiko.transport:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-12T15:52:20.692 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/paramiko/packet.py", line 308, in read_all
2026-02-12T15:52:20.693 ERROR:paramiko.transport:    x = self.__socket.recv(n)
2026-02-12T15:52:20.693 ERROR:paramiko.transport:        ^^^^^^^^^^^^^^^^^^^^^
2026-02-12T15:52:20.693 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 666, in recv
2026-02-12T15:52:20.693 ERROR:paramiko.transport:    self._wait(self._read_event)
2026-02-12T15:52:20.693 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
2026-02-12T15:52:20.693 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 321, in gevent._gevent_c_hub_primitives.wait_on_socket
2026-02-12T15:52:20.693 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: The socket has already been closed by another greenlet
2026-02-12T15:52:20.693 ERROR:paramiko.transport:
2026-02-12T15:52:20.694 INFO:teuthology.orchestra.run.trial094.stdout:NAME="CentOS Stream"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:VERSION="9"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:ID="centos"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:ID_LIKE="rhel fedora"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:VERSION_ID="9"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:PLATFORM_ID="platform:el9"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:ANSI_COLOR="0;31"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:LOGO="fedora-logo-icon"
2026-02-12T15:52:20.695 INFO:teuthology.orchestra.run.trial094.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T15:52:20.696 INFO:teuthology.orchestra.run.trial094.stdout:HOME_URL="https://centos.org/"
2026-02-12T15:52:20.696 INFO:teuthology.orchestra.run.trial094.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T15:52:20.696 INFO:teuthology.orchestra.run.trial094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T15:52:20.696 INFO:teuthology.orchestra.run.trial094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T15:52:20.696 INFO:teuthology.provision.fog.trial094:Deploy complete!
2026-02-12T15:52:20.697 DEBUG:teuthology.parallel:result is None
2026-02-12T15:52:21.221 INFO:teuthology.lock.ops:Checking trial020.front.sepia.ceph.com
2026-02-12T15:52:21.221 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-12T15:52:21.329 INFO:teuthology.lock.ops:Checking trial030.front.sepia.ceph.com
2026-02-12T15:52:21.329 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-12T15:52:21.390 INFO:teuthology.lock.ops:Checking trial094.front.sepia.ceph.com
2026-02-12T15:52:21.390 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-12T15:52:21.446 INFO:teuthology.lock.ops:Updating [trial020.front.sepia.ceph.com]: set os type and version on server
2026-02-12T15:52:21.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial020.front.sepia.ceph.com'
2026-02-12T15:52:21.447 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial020.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:52:21.507 DEBUG:teuthology.orchestra.run.trial020:> true
2026-02-12T15:52:21.584 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial020.front.sepia.ceph.com'
2026-02-12T15:52:21.584 DEBUG:teuthology.orchestra.run.trial020:> uname -m
2026-02-12T15:52:21.639 INFO:teuthology.orchestra.run.trial020.stdout:x86_64
2026-02-12T15:52:21.639 DEBUG:teuthology.orchestra.run.trial020:> cat /etc/os-release
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:NAME="CentOS Stream"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:VERSION="9"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:ID="centos"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:ID_LIKE="rhel fedora"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:VERSION_ID="9"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:PLATFORM_ID="platform:el9"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:ANSI_COLOR="0;31"
2026-02-12T15:52:21.695 INFO:teuthology.orchestra.run.trial020.stdout:LOGO="fedora-logo-icon"
2026-02-12T15:52:21.696 INFO:teuthology.orchestra.run.trial020.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T15:52:21.696 INFO:teuthology.orchestra.run.trial020.stdout:HOME_URL="https://centos.org/"
2026-02-12T15:52:21.696 INFO:teuthology.orchestra.run.trial020.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T15:52:21.696 INFO:teuthology.orchestra.run.trial020.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T15:52:21.696 INFO:teuthology.orchestra.run.trial020.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T15:52:21.697 INFO:teuthology.lock.ops:Updating trial020.front.sepia.ceph.com on lock server
2026-02-12T15:52:21.787 INFO:teuthology.lock.ops:Updating [trial030.front.sepia.ceph.com]: set os type and version on server
2026-02-12T15:52:21.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial030.front.sepia.ceph.com'
2026-02-12T15:52:21.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial030.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:52:21.849 DEBUG:teuthology.orchestra.run.trial030:> true
2026-02-12T15:52:21.931 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial030.front.sepia.ceph.com'
2026-02-12T15:52:21.931 DEBUG:teuthology.orchestra.run.trial030:> uname -m
2026-02-12T15:52:21.985 INFO:teuthology.orchestra.run.trial030.stdout:x86_64
2026-02-12T15:52:21.985 DEBUG:teuthology.orchestra.run.trial030:> cat /etc/os-release
2026-02-12T15:52:22.041 INFO:teuthology.orchestra.run.trial030.stdout:NAME="CentOS Stream"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:VERSION="9"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:ID="centos"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:ID_LIKE="rhel fedora"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:VERSION_ID="9"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:PLATFORM_ID="platform:el9"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:ANSI_COLOR="0;31"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:LOGO="fedora-logo-icon"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:HOME_URL="https://centos.org/"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T15:52:22.042 INFO:teuthology.orchestra.run.trial030.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T15:52:22.042 INFO:teuthology.lock.ops:Updating trial030.front.sepia.ceph.com on lock server
2026-02-12T15:52:22.100 INFO:teuthology.lock.ops:Updating [trial094.front.sepia.ceph.com]: set os type and version on server
2026-02-12T15:52:22.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial094.front.sepia.ceph.com'
2026-02-12T15:52:22.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial094.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T15:52:22.162 DEBUG:teuthology.orchestra.run.trial094:> true
2026-02-12T15:52:22.234 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial094.front.sepia.ceph.com'
2026-02-12T15:52:22.234 DEBUG:teuthology.orchestra.run.trial094:> uname -m
2026-02-12T15:52:22.290 INFO:teuthology.orchestra.run.trial094.stdout:x86_64
2026-02-12T15:52:22.290 DEBUG:teuthology.orchestra.run.trial094:> cat /etc/os-release
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:NAME="CentOS Stream"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:VERSION="9"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:ID="centos"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:ID_LIKE="rhel fedora"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:VERSION_ID="9"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:PLATFORM_ID="platform:el9"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:ANSI_COLOR="0;31"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:LOGO="fedora-logo-icon"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:HOME_URL="https://centos.org/"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T15:52:22.346 INFO:teuthology.orchestra.run.trial094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T15:52:22.346 INFO:teuthology.lock.ops:Updating trial094.front.sepia.ceph.com on lock server
2026-02-12T15:52:22.410 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T15:52:22.504 INFO:teuthology.dispatcher.supervisor:Running job 46375
2026-02-12T15:52:22.504 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_96afe742af26c52260031b90a6912285896e1b3c/virtualenv/bin/teuthology -v --owner scheduled_pdonnell@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/pdonnell-2026-02-12_15:37:21-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/46375 --name pdonnell-2026-02-12_15:37:21-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial --description fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/suites/iogen}} -- /home/teuthworker/mnt/teuthology/pdonnell-2026-02-12_15:37:21-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/46375/orig.config.yaml
2026-02-12T15:52:22.504 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/pdonnell-2026-02-12_15:37:21-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/46375
2026-02-12T15:52:22.504 INFO:teuthology.dispatcher.supervisor:Job PID: 4070238
2026-02-12T15:52:22.504 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-12T15:54:22.605 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T15:56:22.776 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T15:58:22.952 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:00:23.152 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:02:23.339 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:04:23.523 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:06:23.709 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:08:23.810 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:10:23.927 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:12:24.133 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:14:24.306 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:16:24.408 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:18:24.571 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:20:24.725 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T16:20:24.804 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2026-02-12T16:20:24.975 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2026-02-12T16:20:25.701 INFO:teuthology.orchestra.console.trial030:Power off
2026-02-12T16:20:25.701 DEBUG:teuthology.orchestra.console.trial030:pexpect command: ipmitool -H trial030.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T16:20:25.713 INFO:teuthology.orchestra.console.trial094:Power off
2026-02-12T16:20:25.713 DEBUG:teuthology.orchestra.console.trial094:pexpect command: ipmitool -H trial094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T16:20:25.719 INFO:teuthology.orchestra.console.trial020:Power off
2026-02-12T16:20:25.719 DEBUG:teuthology.orchestra.console.trial020:pexpect command: ipmitool -H trial020.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T16:20:25.740 DEBUG:teuthology.orchestra.console.trial030:power off output: Chassis Power Control: Down/Off
2026-02-12T16:20:25.751 DEBUG:teuthology.orchestra.console.trial094:power off output: Chassis Power Control: Down/Off
2026-02-12T16:20:25.757 DEBUG:teuthology.orchestra.console.trial020:power off output: Chassis Power Control: Down/Off
2026-02-12T16:20:29.745 DEBUG:teuthology.orchestra.console.trial030:pexpect command: ipmitool -H trial030.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T16:20:29.753 DEBUG:teuthology.orchestra.console.trial094:pexpect command: ipmitool -H trial094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T16:20:29.759 DEBUG:teuthology.orchestra.console.trial020:pexpect command: ipmitool -H trial020.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T16:20:29.855 DEBUG:teuthology.orchestra.console.trial030:check power output: Chassis Power is on
2026-02-12T16:20:29.861 DEBUG:teuthology.orchestra.console.trial094:check power output: Chassis Power is on
2026-02-12T16:20:29.868 DEBUG:teuthology.orchestra.console.trial020:check power output: Chassis Power is on
2026-02-12T16:20:33.858 DEBUG:teuthology.orchestra.console.trial030:pexpect command: ipmitool -H trial030.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T16:20:33.867 DEBUG:teuthology.orchestra.console.trial094:pexpect command: ipmitool -H trial094.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T16:20:33.875 DEBUG:teuthology.orchestra.console.trial020:pexpect command: ipmitool -H trial020.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T16:20:33.969 DEBUG:teuthology.orchestra.console.trial030:check power output: Chassis Power is off
2026-02-12T16:20:33.969 INFO:teuthology.orchestra.console.trial030:Power off completed
2026-02-12T16:20:33.976 DEBUG:teuthology.orchestra.console.trial094:check power output: Chassis Power is off
2026-02-12T16:20:33.976 INFO:teuthology.orchestra.console.trial094:Power off completed
2026-02-12T16:20:33.984 DEBUG:teuthology.orchestra.console.trial020:check power output: Chassis Power is off
2026-02-12T16:20:33.984 INFO:teuthology.orchestra.console.trial020:Power off completed
2026-02-12T16:20:34.260 INFO:teuthology.lock.ops:unlocked: trial030.front.sepia.ceph.com
2026-02-12T16:20:34.271 INFO:teuthology.lock.ops:unlocked: trial094.front.sepia.ceph.com
2026-02-12T16:20:34.275 INFO:teuthology.lock.ops:unlocked: trial020.front.sepia.ceph.com