2026-02-23T08:33:25.657 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-23T08:33:25.658 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:33:25.731 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:33:26.153 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7619b6a00d40>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7619b627f9c0>, signals=[15])
2026-02-23T08:33:26.153 INFO:teuthology.lock.ops:Start node 'trial038.front.sepia.ceph.com' reimaging
2026-02-23T08:33:26.153 INFO:teuthology.lock.ops:Updating [trial038.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T08:33:26.153 INFO:teuthology.lock.ops:Updating trial038.front.sepia.ceph.com on lock server
2026-02-23T08:33:26.212 INFO:teuthology.lock.ops:Start node 'trial144.front.sepia.ceph.com' reimaging
2026-02-23T08:33:26.212 INFO:teuthology.lock.ops:Updating [trial144.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T08:33:26.212 INFO:teuthology.lock.ops:Updating trial144.front.sepia.ceph.com on lock server
2026-02-23T08:33:26.272 INFO:teuthology.lock.ops:Start node 'trial153.front.sepia.ceph.com' reimaging
2026-02-23T08:33:26.272 INFO:teuthology.lock.ops:Updating [trial153.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T08:33:26.272 INFO:teuthology.lock.ops:Updating trial153.front.sepia.ceph.com on lock server
2026-02-23T08:33:26.507 DEBUG:teuthology.provision.fog.trial144:Requesting image trial_centos_9.stream (ID 3)
2026-02-23T08:33:26.507 DEBUG:teuthology.provision.fog.trial038:Requesting image trial_centos_9.stream (ID 3)
2026-02-23T08:33:26.535 INFO:teuthology.provision.fog.trial038:Scheduling deploy of centos 9.stream
2026-02-23T08:33:26.554 INFO:teuthology.provision.fog.trial144:Scheduling deploy of centos 9.stream
2026-02-23T08:33:26.565 DEBUG:teuthology.provision.fog.trial153:Requesting image trial_centos_9.stream (ID 3)
2026-02-23T08:33:26.594 INFO:teuthology.provision.fog.trial153:Scheduling deploy of centos 9.stream
2026-02-23T08:33:27.902 INFO:teuthology.orchestra.console.trial153:Power off
2026-02-23T08:33:27.903 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T08:33:27.962 INFO:teuthology.orchestra.console.trial144:Power off
2026-02-23T08:33:27.962 DEBUG:teuthology.orchestra.console.trial144:pexpect command: ipmitool -H trial144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T08:33:27.968 DEBUG:teuthology.orchestra.console.trial153:power off output: Chassis Power Control: Down/Off
2026-02-23T08:33:27.982 INFO:teuthology.orchestra.console.trial038:Power off
2026-02-23T08:33:27.982 DEBUG:teuthology.orchestra.console.trial038:pexpect command: ipmitool -H trial038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T08:33:28.000 DEBUG:teuthology.orchestra.console.trial144:power off output: Chassis Power Control: Down/Off
2026-02-23T08:33:28.024 DEBUG:teuthology.orchestra.console.trial038:power off output: Chassis Power Control: Down/Off
2026-02-23T08:33:31.972 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T08:33:32.000 DEBUG:teuthology.orchestra.console.trial144:pexpect command: ipmitool -H trial144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T08:33:32.024 DEBUG:teuthology.orchestra.console.trial038:pexpect command: ipmitool -H trial038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T08:33:32.082 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is off
2026-02-23T08:33:32.082 INFO:teuthology.orchestra.console.trial153:Power off completed
2026-02-23T08:33:32.109 DEBUG:teuthology.orchestra.console.trial144:check power output: Chassis Power is off
2026-02-23T08:33:32.110 INFO:teuthology.orchestra.console.trial144:Power off completed
2026-02-23T08:33:32.134 DEBUG:teuthology.orchestra.console.trial038:check power output: Chassis Power is off
2026-02-23T08:33:32.134 INFO:teuthology.orchestra.console.trial038:Power off completed
2026-02-23T08:33:32.183 INFO:teuthology.orchestra.console.trial153:Power on
2026-02-23T08:33:32.183 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T08:33:32.210 INFO:teuthology.orchestra.console.trial144:Power on
2026-02-23T08:33:32.210 DEBUG:teuthology.orchestra.console.trial144:pexpect command: ipmitool -H trial144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T08:33:32.220 DEBUG:teuthology.orchestra.console.trial153:power on output: Chassis Power Control: Up/On
2026-02-23T08:33:32.234 INFO:teuthology.orchestra.console.trial038:Power on
2026-02-23T08:33:32.235 DEBUG:teuthology.orchestra.console.trial038:pexpect command: ipmitool -H trial038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T08:33:32.248 DEBUG:teuthology.orchestra.console.trial144:power on output: Chassis Power Control: Up/On
2026-02-23T08:33:32.274 DEBUG:teuthology.orchestra.console.trial038:power on output: Chassis Power Control: Up/On
2026-02-23T08:33:36.225 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T08:33:36.249 DEBUG:teuthology.orchestra.console.trial144:pexpect command: ipmitool -H trial144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T08:33:36.275 DEBUG:teuthology.orchestra.console.trial038:pexpect command: ipmitool -H trial038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T08:33:36.336 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is off
2026-02-23T08:33:36.359 DEBUG:teuthology.orchestra.console.trial144:check power output: Chassis Power is on
2026-02-23T08:33:36.359 INFO:teuthology.orchestra.console.trial144:Power on completed
2026-02-23T08:33:36.384 DEBUG:teuthology.orchestra.console.trial038:check power output: Chassis Power is on
2026-02-23T08:33:36.384 INFO:teuthology.orchestra.console.trial038:Power on completed
2026-02-23T08:33:36.460 INFO:teuthology.provision.fog.trial144:Waiting for deploy to finish
2026-02-23T08:33:36.485 INFO:teuthology.provision.fog.trial038:Waiting for deploy to finish
2026-02-23T08:33:40.340 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T08:33:40.450 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is on
2026-02-23T08:33:40.451 INFO:teuthology.orchestra.console.trial153:Power on completed
2026-02-23T08:33:40.551 INFO:teuthology.provision.fog.trial153:Waiting for deploy to finish
2026-02-23T08:35:24.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:35:24.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:35:27.744 WARNING:teuthology.provision.fog.trial144:[Errno None] Unable to connect to port 22 on 10.20.193.144
2026-02-23T08:35:28.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:35:33.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:35:36.832 WARNING:teuthology.provision.fog.trial144:[Errno None] Unable to connect to port 22 on 10.20.193.144
2026-02-23T08:35:42.838 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:35:45.919 WARNING:teuthology.provision.fog.trial144:[Errno None] Unable to connect to port 22 on 10.20.193.144
2026-02-23T08:35:51.926 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:35:55.008 WARNING:teuthology.provision.fog.trial144:[Errno None] Unable to connect to port 22 on 10.20.193.144
2026-02-23T08:36:01.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:36:04.096 WARNING:teuthology.provision.fog.trial144:[Errno None] Unable to connect to port 22 on 10.20.193.144
2026-02-23T08:36:10.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:36:24.647 WARNING:teuthology.provision.fog.trial038:timed out
2026-02-23T08:36:28.022 WARNING:teuthology.provision.fog.trial153:timed out
2026-02-23T08:36:30.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:36:34.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:36:39.311 ERROR:teuthology.orchestra.connection:Error authenticating with trial153.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T08:36:42.011 ERROR:teuthology.orchestra.connection:Error authenticating with trial038.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T08:36:43.407 ERROR:teuthology.orchestra.connection:Error authenticating with trial153.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T08:36:46.091 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T08:36:46.091 DEBUG:teuthology.orchestra.run.trial038:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T08:36:46.152 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T08:36:46.152 DEBUG:teuthology.orchestra.run.trial144:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T08:36:46.385 INFO:teuthology.provision.fog.trial144:Node is ready
2026-02-23T08:36:46.386 DEBUG:teuthology.orchestra.run.trial144:> hostname
2026-02-23T08:36:46.442 INFO:teuthology.orchestra.run.trial144.stdout:trial144.front.sepia.ceph.com
2026-02-23T08:36:46.442 DEBUG:teuthology.orchestra.run.trial144:> grep trial144.front.sepia.ceph.com /etc/hosts
2026-02-23T08:36:46.501 INFO:teuthology.orchestra.run.trial144.stdout:10.20.193.144 trial144.front.sepia.ceph.com trial144
2026-02-23T08:36:46.502 DEBUG:teuthology.orchestra.run.trial144:> sudo hostname trial144
2026-02-23T08:36:46.573 DEBUG:teuthology.orchestra.run.trial144:> sudo sed -i -e 's/trial144.front.sepia.ceph.com/trial144/g' /etc/hosts
2026-02-23T08:36:46.640 DEBUG:teuthology.orchestra.run.trial144:> sudo sed -i -e 's/10.20.193.144/10.20.193.144/g' /etc/hosts
2026-02-23T08:36:46.704 DEBUG:teuthology.orchestra.run.trial144:> sudo sed -i -e 's/trial144.front.sepia.ceph.com/trial144/g' /etc/hostname
2026-02-23T08:36:46.771 DEBUG:teuthology.orchestra.run.trial144:> sudo hostname trial144
2026-02-23T08:36:46.837 DEBUG:teuthology.orchestra.run.trial144:> cat /etc/os-release
2026-02-23T08:36:46.895 INFO:teuthology.orchestra.run.trial144.stdout:NAME="CentOS Stream"
2026-02-23T08:36:46.895 INFO:teuthology.orchestra.run.trial144.stdout:VERSION="9"
2026-02-23T08:36:46.895 INFO:teuthology.orchestra.run.trial144.stdout:ID="centos"
2026-02-23T08:36:46.895 INFO:teuthology.orchestra.run.trial144.stdout:ID_LIKE="rhel fedora"
2026-02-23T08:36:46.895 INFO:teuthology.orchestra.run.trial144.stdout:VERSION_ID="9"
2026-02-23T08:36:46.895 INFO:teuthology.orchestra.run.trial144.stdout:PLATFORM_ID="platform:el9"
2026-02-23T08:36:46.895 INFO:teuthology.orchestra.run.trial144.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T08:36:46.896 INFO:teuthology.orchestra.run.trial144.stdout:ANSI_COLOR="0;31"
2026-02-23T08:36:46.896 INFO:teuthology.orchestra.run.trial144.stdout:LOGO="fedora-logo-icon"
2026-02-23T08:36:46.896 INFO:teuthology.orchestra.run.trial144.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T08:36:46.896 INFO:teuthology.orchestra.run.trial144.stdout:HOME_URL="https://centos.org/"
2026-02-23T08:36:46.896 INFO:teuthology.orchestra.run.trial144.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T08:36:46.896 INFO:teuthology.orchestra.run.trial144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T08:36:46.896 INFO:teuthology.orchestra.run.trial144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T08:36:46.897 INFO:teuthology.provision.fog.trial144:Deploy complete!
2026-02-23T08:36:46.899 DEBUG:teuthology.parallel:result is None
2026-02-23T08:36:50.528 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T08:36:50.528 DEBUG:teuthology.orchestra.run.trial153:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T08:36:50.663 INFO:teuthology.provision.fog.trial153:Node is ready
2026-02-23T08:36:50.664 DEBUG:teuthology.orchestra.run.trial153:> hostname
2026-02-23T08:36:50.721 INFO:teuthology.orchestra.run.trial153.stdout:trial153.front.sepia.ceph.com
2026-02-23T08:36:50.721 DEBUG:teuthology.orchestra.run.trial153:> grep trial153.front.sepia.ceph.com /etc/hosts
2026-02-23T08:36:50.778 INFO:teuthology.orchestra.run.trial153.stdout:10.20.193.153 trial153.front.sepia.ceph.com trial153
2026-02-23T08:36:50.779 DEBUG:teuthology.orchestra.run.trial153:> sudo hostname trial153
2026-02-23T08:36:50.860 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i -e 's/trial153.front.sepia.ceph.com/trial153/g' /etc/hosts
2026-02-23T08:36:50.926 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i -e 's/10.20.193.153/10.20.193.153/g' /etc/hosts
2026-02-23T08:36:50.991 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i -e 's/trial153.front.sepia.ceph.com/trial153/g' /etc/hostname
2026-02-23T08:36:51.056 DEBUG:teuthology.orchestra.run.trial153:> sudo hostname trial153
2026-02-23T08:36:51.121 DEBUG:teuthology.orchestra.run.trial153:> cat /etc/os-release
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:NAME="CentOS Stream"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:VERSION="9"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:ID="centos"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:ID_LIKE="rhel fedora"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:VERSION_ID="9"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:PLATFORM_ID="platform:el9"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:ANSI_COLOR="0;31"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:LOGO="fedora-logo-icon"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:HOME_URL="https://centos.org/"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T08:36:51.177 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T08:36:51.177 INFO:teuthology.provision.fog.trial153:Deploy complete!
2026-02-23T08:36:51.178 DEBUG:teuthology.parallel:result is None
2026-02-23T08:36:51.218 INFO:teuthology.provision.fog.trial038:Node is ready
2026-02-23T08:36:51.218 DEBUG:teuthology.orchestra.run.trial038:> hostname
2026-02-23T08:36:51.235 INFO:teuthology.orchestra.run.trial038.stdout:trial038.front.sepia.ceph.com
2026-02-23T08:36:51.235 DEBUG:teuthology.orchestra.run.trial038:> grep trial038.front.sepia.ceph.com /etc/hosts
2026-02-23T08:36:51.291 INFO:teuthology.orchestra.run.trial038.stdout:10.20.193.38 trial038.front.sepia.ceph.com trial038
2026-02-23T08:36:51.291 DEBUG:teuthology.orchestra.run.trial038:> sudo hostname trial038
2026-02-23T08:36:51.363 DEBUG:teuthology.orchestra.run.trial038:> sudo sed -i -e 's/trial038.front.sepia.ceph.com/trial038/g' /etc/hosts
2026-02-23T08:36:51.428 DEBUG:teuthology.orchestra.run.trial038:> sudo sed -i -e 's/10.20.193.38/10.20.193.38/g' /etc/hosts
2026-02-23T08:36:51.493 DEBUG:teuthology.orchestra.run.trial038:> sudo sed -i -e 's/trial038.front.sepia.ceph.com/trial038/g' /etc/hostname
2026-02-23T08:36:51.557 DEBUG:teuthology.orchestra.run.trial038:> sudo hostname trial038
2026-02-23T08:36:51.621 DEBUG:teuthology.orchestra.run.trial038:> cat /etc/os-release
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:NAME="CentOS Stream"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:VERSION="9"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:ID="centos"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:ID_LIKE="rhel fedora"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:VERSION_ID="9"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:PLATFORM_ID="platform:el9"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:ANSI_COLOR="0;31"
2026-02-23T08:36:51.678 INFO:teuthology.orchestra.run.trial038.stdout:LOGO="fedora-logo-icon"
2026-02-23T08:36:51.679 INFO:teuthology.orchestra.run.trial038.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T08:36:51.679 INFO:teuthology.orchestra.run.trial038.stdout:HOME_URL="https://centos.org/"
2026-02-23T08:36:51.679 INFO:teuthology.orchestra.run.trial038.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T08:36:51.679 INFO:teuthology.orchestra.run.trial038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T08:36:51.679 INFO:teuthology.orchestra.run.trial038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T08:36:51.679 INFO:teuthology.provision.fog.trial038:Deploy complete!
2026-02-23T08:36:51.680 DEBUG:teuthology.parallel:result is None
2026-02-23T08:36:52.193 INFO:teuthology.lock.ops:Checking trial038.front.sepia.ceph.com
2026-02-23T08:36:52.193 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T08:36:52.301 INFO:teuthology.lock.ops:Checking trial144.front.sepia.ceph.com
2026-02-23T08:36:52.301 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T08:36:52.364 INFO:teuthology.lock.ops:Checking trial153.front.sepia.ceph.com
2026-02-23T08:36:52.364 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T08:36:52.423 INFO:teuthology.lock.ops:Updating [trial038.front.sepia.ceph.com]: set os type and version on server
2026-02-23T08:36:52.423 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial038.front.sepia.ceph.com'
2026-02-23T08:36:52.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:36:52.485 DEBUG:teuthology.orchestra.run.trial038:> true
2026-02-23T08:36:52.564 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial038.front.sepia.ceph.com'
2026-02-23T08:36:52.564 DEBUG:teuthology.orchestra.run.trial038:> uname -m
2026-02-23T08:36:52.619 INFO:teuthology.orchestra.run.trial038.stdout:x86_64
2026-02-23T08:36:52.620 DEBUG:teuthology.orchestra.run.trial038:> cat /etc/os-release
2026-02-23T08:36:52.676 INFO:teuthology.orchestra.run.trial038.stdout:NAME="CentOS Stream"
2026-02-23T08:36:52.676 INFO:teuthology.orchestra.run.trial038.stdout:VERSION="9"
2026-02-23T08:36:52.676 INFO:teuthology.orchestra.run.trial038.stdout:ID="centos"
2026-02-23T08:36:52.676 INFO:teuthology.orchestra.run.trial038.stdout:ID_LIKE="rhel fedora"
2026-02-23T08:36:52.676 INFO:teuthology.orchestra.run.trial038.stdout:VERSION_ID="9"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:PLATFORM_ID="platform:el9"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:ANSI_COLOR="0;31"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:LOGO="fedora-logo-icon"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:HOME_URL="https://centos.org/"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T08:36:52.677 INFO:teuthology.orchestra.run.trial038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T08:36:52.678 INFO:teuthology.lock.ops:Updating trial038.front.sepia.ceph.com on lock server
2026-02-23T08:36:52.782 INFO:teuthology.lock.ops:Updating [trial144.front.sepia.ceph.com]: set os type and version on server
2026-02-23T08:36:52.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial144.front.sepia.ceph.com'
2026-02-23T08:36:52.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:36:52.851 DEBUG:teuthology.orchestra.run.trial144:> true
2026-02-23T08:36:52.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial144.front.sepia.ceph.com'
2026-02-23T08:36:52.926 DEBUG:teuthology.orchestra.run.trial144:> uname -m
2026-02-23T08:36:52.982 INFO:teuthology.orchestra.run.trial144.stdout:x86_64
2026-02-23T08:36:52.982 DEBUG:teuthology.orchestra.run.trial144:> cat /etc/os-release
2026-02-23T08:36:53.039 INFO:teuthology.orchestra.run.trial144.stdout:NAME="CentOS Stream"
2026-02-23T08:36:53.039 INFO:teuthology.orchestra.run.trial144.stdout:VERSION="9"
2026-02-23T08:36:53.039 INFO:teuthology.orchestra.run.trial144.stdout:ID="centos"
2026-02-23T08:36:53.039 INFO:teuthology.orchestra.run.trial144.stdout:ID_LIKE="rhel fedora"
2026-02-23T08:36:53.039 INFO:teuthology.orchestra.run.trial144.stdout:VERSION_ID="9"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:PLATFORM_ID="platform:el9"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:ANSI_COLOR="0;31"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:LOGO="fedora-logo-icon"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:HOME_URL="https://centos.org/"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T08:36:53.040 INFO:teuthology.orchestra.run.trial144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T08:36:53.041 INFO:teuthology.lock.ops:Updating trial144.front.sepia.ceph.com on lock server
2026-02-23T08:36:53.143 INFO:teuthology.lock.ops:Updating [trial153.front.sepia.ceph.com]: set os type and version on server
2026-02-23T08:36:53.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial153.front.sepia.ceph.com'
2026-02-23T08:36:53.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T08:36:53.208 DEBUG:teuthology.orchestra.run.trial153:> true
2026-02-23T08:36:53.285 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial153.front.sepia.ceph.com'
2026-02-23T08:36:53.286 DEBUG:teuthology.orchestra.run.trial153:> uname -m
2026-02-23T08:36:53.341 INFO:teuthology.orchestra.run.trial153.stdout:x86_64
2026-02-23T08:36:53.341 DEBUG:teuthology.orchestra.run.trial153:> cat /etc/os-release
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:NAME="CentOS Stream"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:VERSION="9"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:ID="centos"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:ID_LIKE="rhel fedora"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:VERSION_ID="9"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:PLATFORM_ID="platform:el9"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:ANSI_COLOR="0;31"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:LOGO="fedora-logo-icon"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T08:36:53.396 INFO:teuthology.orchestra.run.trial153.stdout:HOME_URL="https://centos.org/"
2026-02-23T08:36:53.397 INFO:teuthology.orchestra.run.trial153.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T08:36:53.397 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T08:36:53.397 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T08:36:53.397 INFO:teuthology.lock.ops:Updating trial153.front.sepia.ceph.com on lock server
2026-02-23T08:36:53.498 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:36:53.593 INFO:teuthology.dispatcher.supervisor:Running job 65709
2026-02-23T08:36:53.593 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_skanta@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/skanta-2026-02-23_07:12:59-rados-wip-bharath8-testing-2026-02-22-0716-distro-default-trial/65709 --name skanta-2026-02-23_07:12:59-rados-wip-bharath8-testing-2026-02-22-0716-distro-default-trial --description rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/tentacle backoff/peering ceph clusters/{three-plus-one} d-balancer/on mon_election/classic msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/radosbench} -- /home/teuthworker/mnt/teuthology/skanta-2026-02-23_07:12:59-rados-wip-bharath8-testing-2026-02-22-0716-distro-default-trial/65709/orig.config.yaml
2026-02-23T08:36:53.594 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/skanta-2026-02-23_07:12:59-rados-wip-bharath8-testing-2026-02-22-0716-distro-default-trial/65709
2026-02-23T08:36:53.594 INFO:teuthology.dispatcher.supervisor:Job PID: 3429930
2026-02-23T08:36:53.594 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-23T08:38:53.694 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:40:53.880 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:42:54.076 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:44:54.261 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:46:54.390 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:48:54.588 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:50:54.810 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:52:55.004 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:54:55.196 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:56:55.323 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T08:58:55.496 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:00:55.684 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:02:55.890 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:04:56.101 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:06:56.236 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:08:56.432 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:10:56.611 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:12:56.808 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:14:57.030 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:16:57.210 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:18:57.353 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:20:57.530 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:22:57.711 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:24:57.807 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:26:57.936 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:28:58.135 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:30:58.303 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:32:58.484 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:34:58.616 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:36:58.808 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:38:58.991 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T09:38:59.079 INFO:teuthology.dispatcher.supervisor:Success!
2026-02-23T09:38:59.250 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2026-02-23T09:38:59.995 INFO:teuthology.orchestra.console.trial153:Power off
2026-02-23T09:38:59.995 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T09:39:00.006 INFO:teuthology.orchestra.console.trial038:Power off
2026-02-23T09:39:00.006 DEBUG:teuthology.orchestra.console.trial038:pexpect command: ipmitool -H trial038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T09:39:00.025 INFO:teuthology.orchestra.console.trial144:Power off
2026-02-23T09:39:00.026 DEBUG:teuthology.orchestra.console.trial144:pexpect command: ipmitool -H trial144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T09:39:00.035 DEBUG:teuthology.orchestra.console.trial153:power off output: Chassis Power Control: Down/Off
2026-02-23T09:39:00.047 DEBUG:teuthology.orchestra.console.trial038:power off output: Chassis Power Control: Down/Off
2026-02-23T09:39:00.064 DEBUG:teuthology.orchestra.console.trial144:power off output: Chassis Power Control: Down/Off
2026-02-23T09:39:04.036 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T09:39:04.048 DEBUG:teuthology.orchestra.console.trial038:pexpect command: ipmitool -H trial038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T09:39:04.064 DEBUG:teuthology.orchestra.console.trial144:pexpect command: ipmitool -H trial144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T09:39:04.145 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is on
2026-02-23T09:39:04.155 DEBUG:teuthology.orchestra.console.trial038:check power output: Chassis Power is on
2026-02-23T09:39:04.170 DEBUG:teuthology.orchestra.console.trial144:check power output: Chassis Power is on
2026-02-23T09:39:08.150 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T09:39:08.161 DEBUG:teuthology.orchestra.console.trial038:pexpect command: ipmitool -H trial038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T09:39:08.171 DEBUG:teuthology.orchestra.console.trial144:pexpect command: ipmitool -H trial144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T09:39:08.263 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is off
2026-02-23T09:39:08.263 INFO:teuthology.orchestra.console.trial153:Power off completed
2026-02-23T09:39:08.272 DEBUG:teuthology.orchestra.console.trial038:check power output: Chassis Power is off
2026-02-23T09:39:08.272 INFO:teuthology.orchestra.console.trial038:Power off completed
2026-02-23T09:39:08.280 DEBUG:teuthology.orchestra.console.trial144:check power output: Chassis Power is off
2026-02-23T09:39:08.280 INFO:teuthology.orchestra.console.trial144:Power off completed
2026-02-23T09:39:08.564 INFO:teuthology.lock.ops:unlocked: trial153.front.sepia.ceph.com
2026-02-23T09:39:08.573 INFO:teuthology.lock.ops:unlocked: trial038.front.sepia.ceph.com
2026-02-23T09:39:08.576 INFO:teuthology.lock.ops:unlocked: trial144.front.sepia.ceph.com