2026-02-20T01:25:56.843 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-20T01:25:56.844 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:25:56.916 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:25:57.173 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7ad9d1ca95e0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7ad9d1484720>, signals=[15])
2026-02-20T01:25:57.173 INFO:teuthology.lock.ops:Start node 'trial047.front.sepia.ceph.com' reimaging
2026-02-20T01:25:57.173 INFO:teuthology.lock.ops:Updating [trial047.front.sepia.ceph.com]: reset os type and version on server
2026-02-20T01:25:57.173 INFO:teuthology.lock.ops:Updating trial047.front.sepia.ceph.com on lock server
2026-02-20T01:25:57.233 INFO:teuthology.lock.ops:Start node 'trial106.front.sepia.ceph.com' reimaging
2026-02-20T01:25:57.233 INFO:teuthology.lock.ops:Updating [trial106.front.sepia.ceph.com]: reset os type and version on server
2026-02-20T01:25:57.233 INFO:teuthology.lock.ops:Updating trial106.front.sepia.ceph.com on lock server
2026-02-20T01:25:57.294 INFO:teuthology.lock.ops:Start node 'trial119.front.sepia.ceph.com' reimaging
2026-02-20T01:25:57.294 INFO:teuthology.lock.ops:Updating [trial119.front.sepia.ceph.com]: reset os type and version on server
2026-02-20T01:25:57.294 INFO:teuthology.lock.ops:Updating trial119.front.sepia.ceph.com on lock server
2026-02-20T01:25:57.527 DEBUG:teuthology.provision.fog.trial106:Requesting image trial_centos_9.stream (ID 3)
2026-02-20T01:25:57.528 DEBUG:teuthology.provision.fog.trial047:Requesting image trial_centos_9.stream (ID 3)
2026-02-20T01:25:57.581 INFO:teuthology.provision.fog.trial106:Scheduling deploy of centos 9.stream
2026-02-20T01:25:57.584 INFO:teuthology.provision.fog.trial047:Scheduling deploy of centos 9.stream
2026-02-20T01:25:57.584 DEBUG:teuthology.provision.fog.trial119:Requesting image trial_centos_9.stream (ID 3)
2026-02-20T01:25:57.638 INFO:teuthology.provision.fog.trial119:Scheduling deploy of centos 9.stream
2026-02-20T01:25:58.857 INFO:teuthology.orchestra.console.trial106:Power off
2026-02-20T01:25:58.857 DEBUG:teuthology.orchestra.console.trial106:pexpect command: ipmitool -H trial106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-20T01:25:58.917 INFO:teuthology.orchestra.console.trial047:Power off
2026-02-20T01:25:58.918 DEBUG:teuthology.orchestra.console.trial047:pexpect command: ipmitool -H trial047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-20T01:25:58.923 DEBUG:teuthology.orchestra.console.trial106:power off output: Chassis Power Control: Down/Off
2026-02-20T01:25:58.939 INFO:teuthology.orchestra.console.trial119:Power off
2026-02-20T01:25:58.939 DEBUG:teuthology.orchestra.console.trial119:pexpect command: ipmitool -H trial119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-20T01:25:58.955 DEBUG:teuthology.orchestra.console.trial047:power off output: Chassis Power Control: Down/Off
2026-02-20T01:25:58.976 DEBUG:teuthology.orchestra.console.trial119:power off output: Chassis Power Control: Down/Off
2026-02-20T01:26:02.928 DEBUG:teuthology.orchestra.console.trial106:pexpect command: ipmitool -H trial106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:26:02.956 DEBUG:teuthology.orchestra.console.trial047:pexpect command: ipmitool -H trial047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:26:02.977 DEBUG:teuthology.orchestra.console.trial119:pexpect command: ipmitool -H trial119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:26:03.037 DEBUG:teuthology.orchestra.console.trial106:check power output: Chassis Power is off
2026-02-20T01:26:03.037 INFO:teuthology.orchestra.console.trial106:Power off completed
2026-02-20T01:26:03.066 DEBUG:teuthology.orchestra.console.trial047:check power output: Chassis Power is off
2026-02-20T01:26:03.066 INFO:teuthology.orchestra.console.trial047:Power off completed
2026-02-20T01:26:03.085 DEBUG:teuthology.orchestra.console.trial119:check power output: Chassis Power is off
2026-02-20T01:26:03.085 INFO:teuthology.orchestra.console.trial119:Power off completed
2026-02-20T01:26:03.138 INFO:teuthology.orchestra.console.trial106:Power on
2026-02-20T01:26:03.138 DEBUG:teuthology.orchestra.console.trial106:pexpect command: ipmitool -H trial106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-20T01:26:03.166 INFO:teuthology.orchestra.console.trial047:Power on
2026-02-20T01:26:03.167 DEBUG:teuthology.orchestra.console.trial047:pexpect command: ipmitool -H trial047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-20T01:26:03.178 DEBUG:teuthology.orchestra.console.trial106:power on output: Chassis Power Control: Up/On
2026-02-20T01:26:03.186 INFO:teuthology.orchestra.console.trial119:Power on
2026-02-20T01:26:03.186 DEBUG:teuthology.orchestra.console.trial119:pexpect command: ipmitool -H trial119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-20T01:26:03.207 DEBUG:teuthology.orchestra.console.trial047:power on output: Chassis Power Control: Up/On
2026-02-20T01:26:03.227 DEBUG:teuthology.orchestra.console.trial119:power on output: Chassis Power Control: Up/On
2026-02-20T01:26:07.184 DEBUG:teuthology.orchestra.console.trial106:pexpect command: ipmitool -H trial106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:26:07.208 DEBUG:teuthology.orchestra.console.trial047:pexpect command: ipmitool -H trial047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:26:07.228 DEBUG:teuthology.orchestra.console.trial119:pexpect command: ipmitool -H trial119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:26:07.296 DEBUG:teuthology.orchestra.console.trial106:check power output: Chassis Power is on
2026-02-20T01:26:07.296 INFO:teuthology.orchestra.console.trial106:Power on completed
2026-02-20T01:26:07.318 DEBUG:teuthology.orchestra.console.trial047:check power output: Chassis Power is on
2026-02-20T01:26:07.318 INFO:teuthology.orchestra.console.trial047:Power on completed
2026-02-20T01:26:07.340 DEBUG:teuthology.orchestra.console.trial119:check power output: Chassis Power is on
2026-02-20T01:26:07.341 INFO:teuthology.orchestra.console.trial119:Power on completed
2026-02-20T01:26:07.397 INFO:teuthology.provision.fog.trial106:Waiting for deploy to finish
2026-02-20T01:26:07.419 INFO:teuthology.provision.fog.trial047:Waiting for deploy to finish
2026-02-20T01:26:07.442 INFO:teuthology.provision.fog.trial119:Waiting for deploy to finish
2026-02-20T01:27:55.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:10.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:10.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:13.568 WARNING:teuthology.provision.fog.trial047:[Errno None] Unable to connect to port 22 on 10.20.193.47
2026-02-20T01:28:19.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:22.656 WARNING:teuthology.provision.fog.trial047:[Errno None] Unable to connect to port 22 on 10.20.193.47
2026-02-20T01:28:24.832 WARNING:teuthology.provision.fog.trial106:[Errno None] Unable to connect to port 22 on 10.20.193.106
2026-02-20T01:28:28.661 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:30.836 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:31.744 WARNING:teuthology.provision.fog.trial047:[Errno None] Unable to connect to port 22 on 10.20.193.47
2026-02-20T01:28:33.920 WARNING:teuthology.provision.fog.trial106:[Errno None] Unable to connect to port 22 on 10.20.193.106
2026-02-20T01:28:37.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:39.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:40.832 WARNING:teuthology.provision.fog.trial047:[Errno None] Unable to connect to port 22 on 10.20.193.47
2026-02-20T01:28:43.008 WARNING:teuthology.provision.fog.trial106:[Errno None] Unable to connect to port 22 on 10.20.193.106
2026-02-20T01:28:46.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:49.012 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:28:55.159 WARNING:teuthology.provision.fog.trial119:timed out
2026-02-20T01:29:01.166 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:29:08.486 ERROR:teuthology.orchestra.connection:Error authenticating with trial119.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-20T01:29:12.574 ERROR:teuthology.orchestra.connection:Error authenticating with trial119.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-20T01:29:19.659 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-20T01:29:19.660 DEBUG:teuthology.orchestra.run.trial119:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-20T01:29:19.799 INFO:teuthology.provision.fog.trial119:Node is ready
2026-02-20T01:29:19.799 DEBUG:teuthology.orchestra.run.trial119:> hostname
2026-02-20T01:29:19.859 INFO:teuthology.orchestra.run.trial119.stdout:trial119.front.sepia.ceph.com
2026-02-20T01:29:19.859 DEBUG:teuthology.orchestra.run.trial119:> grep trial119.front.sepia.ceph.com /etc/hosts
2026-02-20T01:29:19.919 INFO:teuthology.orchestra.run.trial119.stdout:10.20.193.119 trial119.front.sepia.ceph.com trial119
2026-02-20T01:29:19.919 DEBUG:teuthology.orchestra.run.trial119:> sudo hostname trial119
2026-02-20T01:29:19.997 DEBUG:teuthology.orchestra.run.trial119:> sudo sed -i -e 's/trial119.front.sepia.ceph.com/trial119/g' /etc/hosts
2026-02-20T01:29:20.065 DEBUG:teuthology.orchestra.run.trial119:> sudo sed -i -e 's/10.20.193.119/10.20.193.119/g' /etc/hosts
2026-02-20T01:29:20.134 DEBUG:teuthology.orchestra.run.trial119:> sudo sed -i -e 's/trial119.front.sepia.ceph.com/trial119/g' /etc/hostname
2026-02-20T01:29:20.202 DEBUG:teuthology.orchestra.run.trial119:> sudo hostname trial119
2026-02-20T01:29:20.270 DEBUG:teuthology.orchestra.run.trial119:> cat /etc/os-release
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:NAME="CentOS Stream"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:VERSION="9"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:ID="centos"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:ID_LIKE="rhel fedora"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:VERSION_ID="9"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:PLATFORM_ID="platform:el9"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:ANSI_COLOR="0;31"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:LOGO="fedora-logo-icon"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:HOME_URL="https://centos.org/"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-20T01:29:20.329 INFO:teuthology.orchestra.run.trial119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-20T01:29:20.331 INFO:teuthology.provision.fog.trial119:Deploy complete!
2026-02-20T01:29:20.331 DEBUG:teuthology.parallel:result is None
2026-02-20T01:29:22.659 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-20T01:29:22.659 DEBUG:teuthology.orchestra.run.trial047:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-20T01:29:22.812 INFO:teuthology.provision.fog.trial047:Node is ready
2026-02-20T01:29:22.812 DEBUG:teuthology.orchestra.run.trial047:> hostname
2026-02-20T01:29:22.868 INFO:teuthology.orchestra.run.trial047.stdout:trial047.front.sepia.ceph.com
2026-02-20T01:29:22.868 DEBUG:teuthology.orchestra.run.trial047:> grep trial047.front.sepia.ceph.com /etc/hosts
2026-02-20T01:29:22.923 INFO:teuthology.orchestra.run.trial047.stdout:10.20.193.47 trial047.front.sepia.ceph.com trial047
2026-02-20T01:29:22.923 DEBUG:teuthology.orchestra.run.trial047:> sudo hostname trial047
2026-02-20T01:29:22.993 DEBUG:teuthology.orchestra.run.trial047:> sudo sed -i -e 's/trial047.front.sepia.ceph.com/trial047/g' /etc/hosts
2026-02-20T01:29:23.057 DEBUG:teuthology.orchestra.run.trial047:> sudo sed -i -e 's/10.20.193.47/10.20.193.47/g' /etc/hosts
2026-02-20T01:29:23.121 DEBUG:teuthology.orchestra.run.trial047:> sudo sed -i -e 's/trial047.front.sepia.ceph.com/trial047/g' /etc/hostname
2026-02-20T01:29:23.185 DEBUG:teuthology.orchestra.run.trial047:> sudo hostname trial047
2026-02-20T01:29:23.250 DEBUG:teuthology.orchestra.run.trial047:> cat /etc/os-release
2026-02-20T01:29:23.307 INFO:teuthology.orchestra.run.trial047.stdout:NAME="CentOS Stream"
2026-02-20T01:29:23.307 INFO:teuthology.orchestra.run.trial047.stdout:VERSION="9"
2026-02-20T01:29:23.307 INFO:teuthology.orchestra.run.trial047.stdout:ID="centos"
2026-02-20T01:29:23.307 INFO:teuthology.orchestra.run.trial047.stdout:ID_LIKE="rhel fedora"
2026-02-20T01:29:23.307 INFO:teuthology.orchestra.run.trial047.stdout:VERSION_ID="9"
2026-02-20T01:29:23.307 INFO:teuthology.orchestra.run.trial047.stdout:PLATFORM_ID="platform:el9"
2026-02-20T01:29:23.307 INFO:teuthology.orchestra.run.trial047.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-20T01:29:23.308 INFO:teuthology.orchestra.run.trial047.stdout:ANSI_COLOR="0;31"
2026-02-20T01:29:23.308 INFO:teuthology.orchestra.run.trial047.stdout:LOGO="fedora-logo-icon"
2026-02-20T01:29:23.308 INFO:teuthology.orchestra.run.trial047.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-20T01:29:23.308 INFO:teuthology.orchestra.run.trial047.stdout:HOME_URL="https://centos.org/"
2026-02-20T01:29:23.308 INFO:teuthology.orchestra.run.trial047.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-20T01:29:23.308 INFO:teuthology.orchestra.run.trial047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-20T01:29:23.308 INFO:teuthology.orchestra.run.trial047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-20T01:29:23.309 INFO:teuthology.provision.fog.trial047:Deploy complete!
2026-02-20T01:29:23.310 DEBUG:teuthology.parallel:result is None
2026-02-20T01:29:25.241 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-20T01:29:25.241 DEBUG:teuthology.orchestra.run.trial106:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-20T01:29:25.401 INFO:teuthology.provision.fog.trial106:Node is ready
2026-02-20T01:29:25.401 DEBUG:teuthology.orchestra.run.trial106:> hostname
2026-02-20T01:29:25.456 INFO:teuthology.orchestra.run.trial106.stdout:trial106.front.sepia.ceph.com
2026-02-20T01:29:25.457 DEBUG:teuthology.orchestra.run.trial106:> grep trial106.front.sepia.ceph.com /etc/hosts
2026-02-20T01:29:25.512 INFO:teuthology.orchestra.run.trial106.stdout:10.20.193.106 trial106.front.sepia.ceph.com trial106
2026-02-20T01:29:25.513 DEBUG:teuthology.orchestra.run.trial106:> sudo hostname trial106
2026-02-20T01:29:25.583 DEBUG:teuthology.orchestra.run.trial106:> sudo sed -i -e 's/trial106.front.sepia.ceph.com/trial106/g' /etc/hosts
2026-02-20T01:29:25.648 DEBUG:teuthology.orchestra.run.trial106:> sudo sed -i -e 's/10.20.193.106/10.20.193.106/g' /etc/hosts
2026-02-20T01:29:25.713 DEBUG:teuthology.orchestra.run.trial106:> sudo sed -i -e 's/trial106.front.sepia.ceph.com/trial106/g' /etc/hostname
2026-02-20T01:29:25.778 DEBUG:teuthology.orchestra.run.trial106:> sudo hostname trial106
2026-02-20T01:29:25.826 ERROR:paramiko.transport:Unknown exception: The socket has already been closed by another greenlet
2026-02-20T01:29:25.829 ERROR:paramiko.transport:Traceback (most recent call last):
2026-02-20T01:29:25.829 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 2159, in run
2026-02-20T01:29:25.830 ERROR:paramiko.transport:    ptype, m = self.packetizer.read_message()
2026-02-20T01:29:25.830 ERROR:paramiko.transport:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-20T01:29:25.830 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/packet.py", line 463, in read_message
2026-02-20T01:29:25.830 ERROR:paramiko.transport:    header = self.read_all(self.__block_size_in, check_rekey=True)
2026-02-20T01:29:25.830 ERROR:paramiko.transport:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-20T01:29:25.830 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/packet.py", line 308, in read_all
2026-02-20T01:29:25.830 ERROR:paramiko.transport:    x = self.__socket.recv(n)
2026-02-20T01:29:25.830 ERROR:paramiko.transport:        ^^^^^^^^^^^^^^^^^^^^^
2026-02-20T01:29:25.830 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 666, in recv
2026-02-20T01:29:25.830 ERROR:paramiko.transport:    self._wait(self._read_event)
2026-02-20T01:29:25.830 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
2026-02-20T01:29:25.830 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 321, in gevent._gevent_c_hub_primitives.wait_on_socket
2026-02-20T01:29:25.831 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: The socket has already been closed by another greenlet
2026-02-20T01:29:25.831 ERROR:paramiko.transport:
2026-02-20T01:29:25.842 DEBUG:teuthology.orchestra.run.trial106:> cat /etc/os-release
2026-02-20T01:29:25.897 INFO:teuthology.orchestra.run.trial106.stdout:NAME="CentOS Stream"
2026-02-20T01:29:25.897 INFO:teuthology.orchestra.run.trial106.stdout:VERSION="9"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:ID="centos"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:ID_LIKE="rhel fedora"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:VERSION_ID="9"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:PLATFORM_ID="platform:el9"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:ANSI_COLOR="0;31"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:LOGO="fedora-logo-icon"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:HOME_URL="https://centos.org/"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-20T01:29:25.898 INFO:teuthology.orchestra.run.trial106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-20T01:29:25.899 INFO:teuthology.orchestra.run.trial106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-20T01:29:25.899 INFO:teuthology.provision.fog.trial106:Deploy complete!
2026-02-20T01:29:25.900 DEBUG:teuthology.parallel:result is None
2026-02-20T01:29:26.441 INFO:teuthology.lock.ops:Checking trial047.front.sepia.ceph.com
2026-02-20T01:29:26.441 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-20T01:29:26.498 INFO:teuthology.lock.ops:Checking trial106.front.sepia.ceph.com
2026-02-20T01:29:26.498 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-20T01:29:28.242 INFO:teuthology.lock.ops:Checking trial119.front.sepia.ceph.com
2026-02-20T01:29:28.242 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-20T01:29:28.300 INFO:teuthology.lock.ops:Updating [trial047.front.sepia.ceph.com]: set os type and version on server
2026-02-20T01:29:28.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial047.front.sepia.ceph.com'
2026-02-20T01:29:28.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial047.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:29:28.367 DEBUG:teuthology.orchestra.run.trial047:> true
2026-02-20T01:29:28.442 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial047.front.sepia.ceph.com'
2026-02-20T01:29:28.442 DEBUG:teuthology.orchestra.run.trial047:> uname -m
2026-02-20T01:29:28.497 INFO:teuthology.orchestra.run.trial047.stdout:x86_64
2026-02-20T01:29:28.498 DEBUG:teuthology.orchestra.run.trial047:> cat /etc/os-release
2026-02-20T01:29:28.553 INFO:teuthology.orchestra.run.trial047.stdout:NAME="CentOS Stream"
2026-02-20T01:29:28.553 INFO:teuthology.orchestra.run.trial047.stdout:VERSION="9"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:ID="centos"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:ID_LIKE="rhel fedora"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:VERSION_ID="9"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:PLATFORM_ID="platform:el9"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:ANSI_COLOR="0;31"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:LOGO="fedora-logo-icon"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:HOME_URL="https://centos.org/"
2026-02-20T01:29:28.554 INFO:teuthology.orchestra.run.trial047.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-20T01:29:28.555 INFO:teuthology.orchestra.run.trial047.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-20T01:29:28.555 INFO:teuthology.orchestra.run.trial047.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-20T01:29:28.555 INFO:teuthology.lock.ops:Updating trial047.front.sepia.ceph.com on lock server
2026-02-20T01:29:28.662 INFO:teuthology.lock.ops:Updating [trial106.front.sepia.ceph.com]: set os type and version on server
2026-02-20T01:29:28.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial106.front.sepia.ceph.com'
2026-02-20T01:29:28.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial106.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:29:28.740 DEBUG:teuthology.orchestra.run.trial106:> true
2026-02-20T01:29:28.813 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial106.front.sepia.ceph.com'
2026-02-20T01:29:28.813 DEBUG:teuthology.orchestra.run.trial106:> uname -m
2026-02-20T01:29:28.871 INFO:teuthology.orchestra.run.trial106.stdout:x86_64
2026-02-20T01:29:28.872 DEBUG:teuthology.orchestra.run.trial106:> cat /etc/os-release
2026-02-20T01:29:28.936 INFO:teuthology.orchestra.run.trial106.stdout:NAME="CentOS Stream"
2026-02-20T01:29:28.936 INFO:teuthology.orchestra.run.trial106.stdout:VERSION="9"
2026-02-20T01:29:28.936 INFO:teuthology.orchestra.run.trial106.stdout:ID="centos"
2026-02-20T01:29:28.936 INFO:teuthology.orchestra.run.trial106.stdout:ID_LIKE="rhel fedora"
2026-02-20T01:29:28.936 INFO:teuthology.orchestra.run.trial106.stdout:VERSION_ID="9"
2026-02-20T01:29:28.936 INFO:teuthology.orchestra.run.trial106.stdout:PLATFORM_ID="platform:el9"
2026-02-20T01:29:28.936 INFO:teuthology.orchestra.run.trial106.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-20T01:29:28.937 INFO:teuthology.orchestra.run.trial106.stdout:ANSI_COLOR="0;31"
2026-02-20T01:29:28.937 INFO:teuthology.orchestra.run.trial106.stdout:LOGO="fedora-logo-icon"
2026-02-20T01:29:28.937 INFO:teuthology.orchestra.run.trial106.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-20T01:29:28.937 INFO:teuthology.orchestra.run.trial106.stdout:HOME_URL="https://centos.org/"
2026-02-20T01:29:28.937 INFO:teuthology.orchestra.run.trial106.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-20T01:29:28.937 INFO:teuthology.orchestra.run.trial106.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-20T01:29:28.937 INFO:teuthology.orchestra.run.trial106.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-20T01:29:28.938 INFO:teuthology.lock.ops:Updating trial106.front.sepia.ceph.com on lock server
2026-02-20T01:29:29.005 INFO:teuthology.lock.ops:Updating [trial119.front.sepia.ceph.com]: set os type and version on server
2026-02-20T01:29:29.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial119.front.sepia.ceph.com'
2026-02-20T01:29:29.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-20T01:29:29.068 DEBUG:teuthology.orchestra.run.trial119:> true
2026-02-20T01:29:29.147 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial119.front.sepia.ceph.com'
2026-02-20T01:29:29.147 DEBUG:teuthology.orchestra.run.trial119:> uname -m
2026-02-20T01:29:29.203 INFO:teuthology.orchestra.run.trial119.stdout:x86_64
2026-02-20T01:29:29.203 DEBUG:teuthology.orchestra.run.trial119:> cat /etc/os-release
2026-02-20T01:29:29.259 INFO:teuthology.orchestra.run.trial119.stdout:NAME="CentOS Stream"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:VERSION="9"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:ID="centos"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:ID_LIKE="rhel fedora"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:VERSION_ID="9"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:PLATFORM_ID="platform:el9"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:ANSI_COLOR="0;31"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:LOGO="fedora-logo-icon"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:HOME_URL="https://centos.org/"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-20T01:29:29.260 INFO:teuthology.orchestra.run.trial119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-20T01:29:29.261 INFO:teuthology.lock.ops:Updating trial119.front.sepia.ceph.com on lock server
2026-02-20T01:29:29.355 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:29:29.446 INFO:teuthology.dispatcher.supervisor:Running job 55271
2026-02-20T01:29:29.446 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_jcollin@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/jcollin-2026-02-17_23:11:18-fs-wip-jcollin-testing-20260217.130836-squid-distro-default-trial/55271 --name jcollin-2026-02-17_23:11:18-fs-wip-jcollin-testing-20260217.130836-squid-distro-default-trial --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-kernel 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} -- /home/teuthworker/mnt/teuthology/jcollin-2026-02-17_23:11:18-fs-wip-jcollin-testing-20260217.130836-squid-distro-default-trial/55271/orig.config.yaml
2026-02-20T01:29:29.446 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/jcollin-2026-02-17_23:11:18-fs-wip-jcollin-testing-20260217.130836-squid-distro-default-trial/55271
2026-02-20T01:29:29.446 INFO:teuthology.dispatcher.supervisor:Job PID: 3432254
2026-02-20T01:29:29.447 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-20T01:31:29.546 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:33:29.741 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:35:29.947 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:37:30.157 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:39:30.369 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:41:30.544 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:43:30.736 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-20T01:43:30.814 INFO:teuthology.dispatcher.supervisor:Success!
2026-02-20T01:43:30.993 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2026-02-20T01:43:31.769 INFO:teuthology.orchestra.console.trial119:Power off
2026-02-20T01:43:31.770 DEBUG:teuthology.orchestra.console.trial119:pexpect command: ipmitool -H trial119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-20T01:43:31.783 INFO:teuthology.orchestra.console.trial106:Power off
2026-02-20T01:43:31.783 DEBUG:teuthology.orchestra.console.trial106:pexpect command: ipmitool -H trial106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-20T01:43:31.791 INFO:teuthology.orchestra.console.trial047:Power off
2026-02-20T01:43:31.791 DEBUG:teuthology.orchestra.console.trial047:pexpect command: ipmitool -H trial047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-20T01:43:31.809 DEBUG:teuthology.orchestra.console.trial119:power off output: Chassis Power Control: Down/Off
2026-02-20T01:43:31.821 DEBUG:teuthology.orchestra.console.trial106:power off output: Chassis Power Control: Down/Off
2026-02-20T01:43:31.829 DEBUG:teuthology.orchestra.console.trial047:power off output: Chassis Power Control: Down/Off
2026-02-20T01:43:35.814 DEBUG:teuthology.orchestra.console.trial119:pexpect command: ipmitool -H trial119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:43:35.822 DEBUG:teuthology.orchestra.console.trial106:pexpect command: ipmitool -H trial106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:43:35.830 DEBUG:teuthology.orchestra.console.trial047:pexpect command: ipmitool -H trial047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:43:35.924 DEBUG:teuthology.orchestra.console.trial119:check power output: Chassis Power is on
2026-02-20T01:43:35.931 DEBUG:teuthology.orchestra.console.trial106:check power output: Chassis Power is on
2026-02-20T01:43:35.938 DEBUG:teuthology.orchestra.console.trial047:check power output: Chassis Power is on
2026-02-20T01:43:39.929 DEBUG:teuthology.orchestra.console.trial119:pexpect command: ipmitool -H trial119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:43:39.937 DEBUG:teuthology.orchestra.console.trial106:pexpect command: ipmitool -H trial106.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:43:39.943 DEBUG:teuthology.orchestra.console.trial047:pexpect command: ipmitool -H trial047.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-20T01:43:40.039 DEBUG:teuthology.orchestra.console.trial119:check power output: Chassis Power is off
2026-02-20T01:43:40.040 INFO:teuthology.orchestra.console.trial119:Power off completed
2026-02-20T01:43:40.045 DEBUG:teuthology.orchestra.console.trial106:check power output: Chassis Power is off
2026-02-20T01:43:40.045 INFO:teuthology.orchestra.console.trial106:Power off completed
2026-02-20T01:43:40.051 DEBUG:teuthology.orchestra.console.trial047:check power output: Chassis Power is off
2026-02-20T01:43:40.051 INFO:teuthology.orchestra.console.trial047:Power off completed
2026-02-20T01:43:40.332 INFO:teuthology.lock.ops:unlocked: trial106.front.sepia.ceph.com
2026-02-20T01:43:40.332 INFO:teuthology.lock.ops:unlocked: trial119.front.sepia.ceph.com
2026-02-20T01:43:40.347 INFO:teuthology.lock.ops:unlocked: trial047.front.sepia.ceph.com