2026-02-23T17:50:08.986 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-23T17:50:08.986 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T17:50:09.057 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T17:50:09.356 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7ea43c72b740>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7ea43bf1fd80>, signals=[15])
2026-02-23T17:50:09.356 INFO:teuthology.lock.ops:Start node 'trial041.front.sepia.ceph.com' reimaging
2026-02-23T17:50:09.356 INFO:teuthology.lock.ops:Updating [trial041.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T17:50:09.356 INFO:teuthology.lock.ops:Updating trial041.front.sepia.ceph.com on lock server
2026-02-23T17:50:09.410 INFO:teuthology.lock.ops:Start node 'trial078.front.sepia.ceph.com' reimaging
2026-02-23T17:50:09.410 INFO:teuthology.lock.ops:Updating [trial078.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T17:50:09.410 INFO:teuthology.lock.ops:Updating trial078.front.sepia.ceph.com on lock server
2026-02-23T17:50:09.467 INFO:teuthology.lock.ops:Start node 'trial097.front.sepia.ceph.com' reimaging
2026-02-23T17:50:09.467 INFO:teuthology.lock.ops:Updating [trial097.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T17:50:09.467 INFO:teuthology.lock.ops:Updating trial097.front.sepia.ceph.com on lock server
2026-02-23T17:50:09.562 INFO:teuthology.lock.ops:Start node 'trial155.front.sepia.ceph.com' reimaging
2026-02-23T17:50:09.562 INFO:teuthology.lock.ops:Updating [trial155.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T17:50:09.562 INFO:teuthology.lock.ops:Updating trial155.front.sepia.ceph.com on lock server
2026-02-23T17:50:09.724 DEBUG:teuthology.provision.fog.trial041:Requesting image trial_rocky_10.1 (ID 4)
2026-02-23T17:50:09.725 DEBUG:teuthology.provision.fog.trial078:Requesting image trial_rocky_10.1 (ID 4)
2026-02-23T17:50:09.773 INFO:teuthology.provision.fog.trial041:Scheduling deploy of rocky 10.1
2026-02-23T17:50:09.805 DEBUG:teuthology.provision.fog.trial097:Requesting image trial_rocky_10.1 (ID 4)
2026-02-23T17:50:09.808 INFO:teuthology.provision.fog.trial078:Scheduling deploy of rocky 10.1
2026-02-23T17:50:09.809 DEBUG:teuthology.provision.fog.trial155:Requesting image trial_rocky_10.1 (ID 4)
2026-02-23T17:50:09.831 INFO:teuthology.provision.fog.trial155:Scheduling deploy of rocky 10.1
2026-02-23T17:50:09.834 INFO:teuthology.provision.fog.trial097:Scheduling deploy of rocky 10.1
2026-02-23T17:50:17.001 INFO:teuthology.orchestra.console.trial155:Power off
2026-02-23T17:50:17.001 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T17:50:17.039 DEBUG:teuthology.orchestra.console.trial155:power off output: Chassis Power Control: Down/Off
2026-02-23T17:50:17.168 INFO:teuthology.orchestra.console.trial078:Power off
2026-02-23T17:50:17.168 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T17:50:17.206 DEBUG:teuthology.orchestra.console.trial078:power off output: Chassis Power Control: Down/Off
2026-02-23T17:50:17.374 INFO:teuthology.orchestra.console.trial097:Power off
2026-02-23T17:50:17.374 DEBUG:teuthology.orchestra.console.trial097:pexpect command: ipmitool -H trial097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T17:50:17.444 DEBUG:teuthology.orchestra.console.trial097:power off output: Chassis Power Control: Down/Off
2026-02-23T17:50:17.456 INFO:teuthology.orchestra.console.trial041:Power off
2026-02-23T17:50:17.456 DEBUG:teuthology.orchestra.console.trial041:pexpect command: ipmitool -H trial041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T17:50:17.493 DEBUG:teuthology.orchestra.console.trial041:power off output: Chassis Power Control: Down/Off
2026-02-23T17:50:21.044 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:21.154 DEBUG:teuthology.orchestra.console.trial155:check power output: Chassis Power is off
2026-02-23T17:50:21.155 INFO:teuthology.orchestra.console.trial155:Power off completed
2026-02-23T17:50:21.207 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:21.256 INFO:teuthology.orchestra.console.trial155:Power on
2026-02-23T17:50:21.256 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T17:50:21.293 DEBUG:teuthology.orchestra.console.trial155:power on output: Chassis Power Control: Up/On
2026-02-23T17:50:21.315 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is off
2026-02-23T17:50:21.316 INFO:teuthology.orchestra.console.trial078:Power off completed
2026-02-23T17:50:21.417 INFO:teuthology.orchestra.console.trial078:Power on
2026-02-23T17:50:21.417 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T17:50:21.445 DEBUG:teuthology.orchestra.console.trial097:pexpect command: ipmitool -H trial097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:21.456 DEBUG:teuthology.orchestra.console.trial078:power on output: Chassis Power Control: Up/On
2026-02-23T17:50:21.495 DEBUG:teuthology.orchestra.console.trial041:pexpect command: ipmitool -H trial041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:21.555 DEBUG:teuthology.orchestra.console.trial097:check power output: Chassis Power is off
2026-02-23T17:50:21.555 INFO:teuthology.orchestra.console.trial097:Power off completed
2026-02-23T17:50:21.604 DEBUG:teuthology.orchestra.console.trial041:check power output: Chassis Power is off
2026-02-23T17:50:21.604 INFO:teuthology.orchestra.console.trial041:Power off completed
2026-02-23T17:50:21.657 INFO:teuthology.orchestra.console.trial097:Power on
2026-02-23T17:50:21.657 DEBUG:teuthology.orchestra.console.trial097:pexpect command: ipmitool -H trial097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T17:50:21.694 DEBUG:teuthology.orchestra.console.trial097:power on output: Chassis Power Control: Up/On
2026-02-23T17:50:21.705 INFO:teuthology.orchestra.console.trial041:Power on
2026-02-23T17:50:21.705 DEBUG:teuthology.orchestra.console.trial041:pexpect command: ipmitool -H trial041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T17:50:21.744 DEBUG:teuthology.orchestra.console.trial041:power on output: Chassis Power Control: Up/On
2026-02-23T17:50:25.298 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:25.409 DEBUG:teuthology.orchestra.console.trial155:check power output: Chassis Power is off
2026-02-23T17:50:25.457 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:25.567 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is off
2026-02-23T17:50:25.695 DEBUG:teuthology.orchestra.console.trial097:pexpect command: ipmitool -H trial097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:25.745 DEBUG:teuthology.orchestra.console.trial041:pexpect command: ipmitool -H trial041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:25.806 DEBUG:teuthology.orchestra.console.trial097:check power output: Chassis Power is on
2026-02-23T17:50:25.806 INFO:teuthology.orchestra.console.trial097:Power on completed
2026-02-23T17:50:25.856 DEBUG:teuthology.orchestra.console.trial041:check power output: Chassis Power is on
2026-02-23T17:50:25.856 INFO:teuthology.orchestra.console.trial041:Power on completed
2026-02-23T17:50:25.907 INFO:teuthology.provision.fog.trial097:Waiting for deploy to finish
2026-02-23T17:50:25.957 INFO:teuthology.provision.fog.trial041:Waiting for deploy to finish
2026-02-23T17:50:29.412 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:29.523 DEBUG:teuthology.orchestra.console.trial155:check power output: Chassis Power is on
2026-02-23T17:50:29.523 INFO:teuthology.orchestra.console.trial155:Power on completed
2026-02-23T17:50:29.568 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T17:50:29.625 INFO:teuthology.provision.fog.trial155:Waiting for deploy to finish
2026-02-23T17:50:29.678 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is on
2026-02-23T17:50:29.678 INFO:teuthology.orchestra.console.trial078:Power on completed
2026-02-23T17:50:29.778 INFO:teuthology.provision.fog.trial078:Waiting for deploy to finish
2026-02-23T17:53:50.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:54:04.416 WARNING:teuthology.provision.fog.trial155:[Errno None] Unable to connect to port 22 on 10.20.193.155
2026-02-23T17:54:10.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:54:13.504 WARNING:teuthology.provision.fog.trial155:[Errno None] Unable to connect to port 22 on 10.20.193.155
2026-02-23T17:54:19.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:54:22.592 WARNING:teuthology.provision.fog.trial155:[Errno None] Unable to connect to port 22 on 10.20.193.155
2026-02-23T17:54:28.598 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:55:04.856 ERROR:teuthology.orchestra.connection:Error authenticating with trial155.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T17:55:08.966 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T17:55:08.966 DEBUG:teuthology.orchestra.run.trial155:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T17:55:09.164 INFO:teuthology.provision.fog.trial155:Node is ready
2026-02-23T17:55:09.164 DEBUG:teuthology.orchestra.run.trial155:> hostname
2026-02-23T17:55:09.222 INFO:teuthology.orchestra.run.trial155.stdout:trial155.front.sepia.ceph.com
2026-02-23T17:55:09.223 DEBUG:teuthology.orchestra.run.trial155:> grep trial155.front.sepia.ceph.com /etc/hosts
2026-02-23T17:55:09.281 INFO:teuthology.orchestra.run.trial155.stdout:10.20.193.155 trial155.front.sepia.ceph.com trial155
2026-02-23T17:55:09.281 DEBUG:teuthology.orchestra.run.trial155:> sudo hostname trial155
2026-02-23T17:55:09.354 DEBUG:teuthology.orchestra.run.trial155:> sudo sed -i -e 's/trial155.front.sepia.ceph.com/trial155/g' /etc/hosts
2026-02-23T17:55:09.421 DEBUG:teuthology.orchestra.run.trial155:> sudo sed -i -e 's/10.20.193.155/10.20.193.155/g' /etc/hosts
2026-02-23T17:55:09.489 DEBUG:teuthology.orchestra.run.trial155:> sudo sed -i -e 's/trial155.front.sepia.ceph.com/trial155/g' /etc/hostname
2026-02-23T17:55:09.556 DEBUG:teuthology.orchestra.run.trial155:> sudo hostname trial155
2026-02-23T17:55:09.624 DEBUG:teuthology.orchestra.run.trial155:> cat /etc/os-release
2026-02-23T17:55:09.683 INFO:teuthology.orchestra.run.trial155.stdout:NAME="Rocky Linux"
2026-02-23T17:55:09.683 INFO:teuthology.orchestra.run.trial155.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:55:09.683 INFO:teuthology.orchestra.run.trial155.stdout:ID="rocky"
2026-02-23T17:55:09.683 INFO:teuthology.orchestra.run.trial155.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:55:09.683 INFO:teuthology.orchestra.run.trial155.stdout:VERSION_ID="10.1"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:ANSI_COLOR="0;32"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:VENDOR_NAME="RESF"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:55:09.684 INFO:teuthology.orchestra.run.trial155.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:55:09.685 INFO:teuthology.orchestra.run.trial155.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:55:09.685 INFO:teuthology.orchestra.run.trial155.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:55:09.685 INFO:teuthology.orchestra.run.trial155.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:55:09.685 INFO:teuthology.orchestra.run.trial155.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:55:09.685 INFO:teuthology.orchestra.run.trial155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:55:09.686 INFO:teuthology.provision.fog.trial155:Deploy complete!
2026-02-23T17:55:09.687 DEBUG:teuthology.parallel:result is None
2026-02-23T17:55:26.706 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:55:29.792 WARNING:teuthology.provision.fog.trial078:[Errno None] Unable to connect to port 22 on 10.20.193.78
2026-02-23T17:55:35.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:55:38.880 WARNING:teuthology.provision.fog.trial078:[Errno None] Unable to connect to port 22 on 10.20.193.78
2026-02-23T17:55:44.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:55:47.967 WARNING:teuthology.provision.fog.trial078:[Errno None] Unable to connect to port 22 on 10.20.193.78
2026-02-23T17:55:53.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:55:53.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:55:56.864 WARNING:teuthology.provision.fog.trial041:[Errno None] Unable to connect to port 22 on 10.20.193.41
2026-02-23T17:55:57.056 WARNING:teuthology.provision.fog.trial078:[Errno None] Unable to connect to port 22 on 10.20.193.78
2026-02-23T17:56:02.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:03.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:05.952 WARNING:teuthology.provision.fog.trial041:[Errno None] Unable to connect to port 22 on 10.20.193.41
2026-02-23T17:56:10.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:11.955 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:13.184 WARNING:teuthology.provision.fog.trial097:[Errno None] Unable to connect to port 22 on 10.20.193.97
2026-02-23T17:56:15.040 WARNING:teuthology.provision.fog.trial041:[Errno None] Unable to connect to port 22 on 10.20.193.41
2026-02-23T17:56:19.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:21.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:22.272 WARNING:teuthology.provision.fog.trial097:[Errno None] Unable to connect to port 22 on 10.20.193.97
2026-02-23T17:56:24.129 WARNING:teuthology.provision.fog.trial041:[Errno None] Unable to connect to port 22 on 10.20.193.41
2026-02-23T17:56:28.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:30.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:31.359 WARNING:teuthology.provision.fog.trial097:[Errno None] Unable to connect to port 22 on 10.20.193.97
2026-02-23T17:56:33.215 WARNING:teuthology.provision.fog.trial041:[Errno None] Unable to connect to port 22 on 10.20.193.41
2026-02-23T17:56:37.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:39.055 ERROR:teuthology.orchestra.connection:Error authenticating with trial078.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T17:56:39.217 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:56:40.448 WARNING:teuthology.provision.fog.trial097:[Errno None] Unable to connect to port 22 on 10.20.193.97
2026-02-23T17:56:43.130 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T17:56:43.130 DEBUG:teuthology.orchestra.run.trial078:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T17:56:43.323 INFO:teuthology.provision.fog.trial078:Node is ready
2026-02-23T17:56:43.323 DEBUG:teuthology.orchestra.run.trial078:> hostname
2026-02-23T17:56:43.377 INFO:teuthology.orchestra.run.trial078.stdout:trial078.front.sepia.ceph.com
2026-02-23T17:56:43.377 DEBUG:teuthology.orchestra.run.trial078:> grep trial078.front.sepia.ceph.com /etc/hosts
2026-02-23T17:56:43.431 INFO:teuthology.orchestra.run.trial078.stdout:10.20.193.78 trial078.front.sepia.ceph.com trial078
2026-02-23T17:56:43.432 DEBUG:teuthology.orchestra.run.trial078:> sudo hostname trial078
2026-02-23T17:56:43.502 DEBUG:teuthology.orchestra.run.trial078:> sudo sed -i -e 's/trial078.front.sepia.ceph.com/trial078/g' /etc/hosts
2026-02-23T17:56:43.566 DEBUG:teuthology.orchestra.run.trial078:> sudo sed -i -e 's/10.20.193.78/10.20.193.78/g' /etc/hosts
2026-02-23T17:56:43.631 DEBUG:teuthology.orchestra.run.trial078:> sudo sed -i -e 's/trial078.front.sepia.ceph.com/trial078/g' /etc/hostname
2026-02-23T17:56:43.694 DEBUG:teuthology.orchestra.run.trial078:> sudo hostname trial078
2026-02-23T17:56:43.758 DEBUG:teuthology.orchestra.run.trial078:> cat /etc/os-release
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:NAME="Rocky Linux"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:ID="rocky"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:VERSION_ID="10.1"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:ANSI_COLOR="0;32"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:VENDOR_NAME="RESF"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:56:43.812 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:56:43.813 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:56:43.813 INFO:teuthology.provision.fog.trial078:Deploy complete!
2026-02-23T17:56:43.813 DEBUG:teuthology.parallel:result is None
2026-02-23T17:56:46.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:57:15.366 ERROR:teuthology.orchestra.connection:Error authenticating with trial041.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T17:57:19.438 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T17:57:19.438 DEBUG:teuthology.orchestra.run.trial041:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T17:57:19.629 INFO:teuthology.provision.fog.trial041:Node is ready
2026-02-23T17:57:19.629 DEBUG:teuthology.orchestra.run.trial041:> hostname
2026-02-23T17:57:19.692 INFO:teuthology.orchestra.run.trial041.stdout:trial041.front.sepia.ceph.com
2026-02-23T17:57:19.693 DEBUG:teuthology.orchestra.run.trial041:> grep trial041.front.sepia.ceph.com /etc/hosts
2026-02-23T17:57:19.750 INFO:teuthology.orchestra.run.trial041.stdout:10.20.193.41 trial041.front.sepia.ceph.com trial041
2026-02-23T17:57:19.750 DEBUG:teuthology.orchestra.run.trial041:> sudo hostname trial041
2026-02-23T17:57:19.822 DEBUG:teuthology.orchestra.run.trial041:> sudo sed -i -e 's/trial041.front.sepia.ceph.com/trial041/g' /etc/hosts
2026-02-23T17:57:19.889 DEBUG:teuthology.orchestra.run.trial041:> sudo sed -i -e 's/10.20.193.41/10.20.193.41/g' /etc/hosts
2026-02-23T17:57:19.956 DEBUG:teuthology.orchestra.run.trial041:> sudo sed -i -e 's/trial041.front.sepia.ceph.com/trial041/g' /etc/hostname
2026-02-23T17:57:20.036 DEBUG:teuthology.orchestra.run.trial041:> sudo hostname trial041
2026-02-23T17:57:20.103 DEBUG:teuthology.orchestra.run.trial041:> cat /etc/os-release
2026-02-23T17:57:20.160 INFO:teuthology.orchestra.run.trial041.stdout:NAME="Rocky Linux"
2026-02-23T17:57:20.160 INFO:teuthology.orchestra.run.trial041.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:57:20.160 INFO:teuthology.orchestra.run.trial041.stdout:ID="rocky"
2026-02-23T17:57:20.160 INFO:teuthology.orchestra.run.trial041.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:57:20.160 INFO:teuthology.orchestra.run.trial041.stdout:VERSION_ID="10.1"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:ANSI_COLOR="0;32"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:VENDOR_NAME="RESF"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:57:20.161 INFO:teuthology.orchestra.run.trial041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:20.161 INFO:teuthology.provision.fog.trial041:Deploy complete!
2026-02-23T17:57:20.161 DEBUG:teuthology.parallel:result is None
2026-02-23T17:57:22.508 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T17:57:22.509 DEBUG:teuthology.orchestra.run.trial097:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T17:57:27.822 INFO:teuthology.provision.fog.trial097:Node is ready
2026-02-23T17:57:27.823 DEBUG:teuthology.orchestra.run.trial097:> hostname
2026-02-23T17:57:27.840 INFO:teuthology.orchestra.run.trial097.stdout:trial097.front.sepia.ceph.com
2026-02-23T17:57:27.840 DEBUG:teuthology.orchestra.run.trial097:> grep trial097.front.sepia.ceph.com /etc/hosts
2026-02-23T17:57:27.897 INFO:teuthology.orchestra.run.trial097.stdout:10.20.193.97 trial097.front.sepia.ceph.com trial097
2026-02-23T17:57:27.898 DEBUG:teuthology.orchestra.run.trial097:> sudo hostname trial097
2026-02-23T17:57:27.970 DEBUG:teuthology.orchestra.run.trial097:> sudo sed -i -e 's/trial097.front.sepia.ceph.com/trial097/g' /etc/hosts
2026-02-23T17:57:28.038 DEBUG:teuthology.orchestra.run.trial097:> sudo sed -i -e 's/10.20.193.97/10.20.193.97/g' /etc/hosts
2026-02-23T17:57:28.105 DEBUG:teuthology.orchestra.run.trial097:> sudo sed -i -e 's/trial097.front.sepia.ceph.com/trial097/g' /etc/hostname
2026-02-23T17:57:28.171 DEBUG:teuthology.orchestra.run.trial097:> sudo hostname trial097
2026-02-23T17:57:28.238 DEBUG:teuthology.orchestra.run.trial097:> cat /etc/os-release
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:NAME="Rocky Linux"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:ID="rocky"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:VERSION_ID="10.1"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:ANSI_COLOR="0;32"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:VENDOR_NAME="RESF"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:57:28.296 INFO:teuthology.orchestra.run.trial097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:28.297 INFO:teuthology.provision.fog.trial097:Deploy complete!
2026-02-23T17:57:28.297 DEBUG:teuthology.parallel:result is None
2026-02-23T17:57:28.348 ERROR:paramiko.transport:Unknown exception: The socket has already been closed by another greenlet
2026-02-23T17:57:28.349 ERROR:paramiko.transport:Traceback (most recent call last):
2026-02-23T17:57:28.349 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 2159, in run
2026-02-23T17:57:28.349 ERROR:paramiko.transport:    ptype, m = self.packetizer.read_message()
2026-02-23T17:57:28.350 ERROR:paramiko.transport:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-23T17:57:28.350 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/packet.py", line 463, in read_message
2026-02-23T17:57:28.350 ERROR:paramiko.transport:    header = self.read_all(self.__block_size_in, check_rekey=True)
2026-02-23T17:57:28.350 ERROR:paramiko.transport:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-02-23T17:57:28.350 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/packet.py", line 308, in read_all
2026-02-23T17:57:28.350 ERROR:paramiko.transport:    x = self.__socket.recv(n)
2026-02-23T17:57:28.350 ERROR:paramiko.transport:        ^^^^^^^^^^^^^^^^^^^^^
2026-02-23T17:57:28.350 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 666, in recv
2026-02-23T17:57:28.350 ERROR:paramiko.transport:    self._wait(self._read_event)
2026-02-23T17:57:28.350 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
2026-02-23T17:57:28.350 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 321, in gevent._gevent_c_hub_primitives.wait_on_socket
2026-02-23T17:57:28.350 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: The socket has already been closed by another greenlet
2026-02-23T17:57:28.350 ERROR:paramiko.transport:
2026-02-23T17:57:28.936 INFO:teuthology.lock.ops:Checking trial041.front.sepia.ceph.com
2026-02-23T17:57:28.936 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T17:57:29.018 INFO:teuthology.lock.ops:Checking trial078.front.sepia.ceph.com
2026-02-23T17:57:29.019 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T17:57:29.077 INFO:teuthology.lock.ops:Checking trial097.front.sepia.ceph.com
2026-02-23T17:57:29.077 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T17:57:29.139 INFO:teuthology.lock.ops:Checking trial155.front.sepia.ceph.com
2026-02-23T17:57:29.139 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T17:57:29.197 INFO:teuthology.lock.ops:Updating [trial041.front.sepia.ceph.com]: set os type and version on server
2026-02-23T17:57:29.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial041.front.sepia.ceph.com'
2026-02-23T17:57:29.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial041.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:57:29.258 DEBUG:teuthology.orchestra.run.trial041:> true
2026-02-23T17:57:29.341 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial041.front.sepia.ceph.com'
2026-02-23T17:57:29.341 DEBUG:teuthology.orchestra.run.trial041:> uname -m
2026-02-23T17:57:29.398 INFO:teuthology.orchestra.run.trial041.stdout:x86_64
2026-02-23T17:57:29.398 DEBUG:teuthology.orchestra.run.trial041:> cat /etc/os-release
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:NAME="Rocky Linux"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:ID="rocky"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:VERSION_ID="10.1"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:ANSI_COLOR="0;32"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:57:29.457 INFO:teuthology.orchestra.run.trial041.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:VENDOR_NAME="RESF"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:57:29.458 INFO:teuthology.orchestra.run.trial041.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:29.458 INFO:teuthology.lock.ops:Updating trial041.front.sepia.ceph.com on lock server
2026-02-23T17:57:29.516 INFO:teuthology.lock.ops:Updating [trial078.front.sepia.ceph.com]: set os type and version on server
2026-02-23T17:57:29.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial078.front.sepia.ceph.com'
2026-02-23T17:57:29.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:57:29.583 DEBUG:teuthology.orchestra.run.trial078:> true
2026-02-23T17:57:29.777 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial078.front.sepia.ceph.com'
2026-02-23T17:57:29.777 DEBUG:teuthology.orchestra.run.trial078:> uname -m
2026-02-23T17:57:29.835 INFO:teuthology.orchestra.run.trial078.stdout:x86_64
2026-02-23T17:57:29.835 DEBUG:teuthology.orchestra.run.trial078:> cat /etc/os-release
2026-02-23T17:57:29.892 INFO:teuthology.orchestra.run.trial078.stdout:NAME="Rocky Linux"
2026-02-23T17:57:29.892 INFO:teuthology.orchestra.run.trial078.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:57:29.892 INFO:teuthology.orchestra.run.trial078.stdout:ID="rocky"
2026-02-23T17:57:29.892 INFO:teuthology.orchestra.run.trial078.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:57:29.892 INFO:teuthology.orchestra.run.trial078.stdout:VERSION_ID="10.1"
2026-02-23T17:57:29.892 INFO:teuthology.orchestra.run.trial078.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:57:29.892 INFO:teuthology.orchestra.run.trial078.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:ANSI_COLOR="0;32"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:VENDOR_NAME="RESF"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:57:29.893 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:29.893 INFO:teuthology.lock.ops:Updating trial078.front.sepia.ceph.com on lock server
2026-02-23T17:57:29.968 INFO:teuthology.lock.ops:Updating [trial097.front.sepia.ceph.com]: set os type and version on server
2026-02-23T17:57:29.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial097.front.sepia.ceph.com'
2026-02-23T17:57:29.969 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:57:30.037 DEBUG:teuthology.orchestra.run.trial097:> true
2026-02-23T17:57:30.123 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial097.front.sepia.ceph.com'
2026-02-23T17:57:30.123 DEBUG:teuthology.orchestra.run.trial097:> uname -m
2026-02-23T17:57:30.180 INFO:teuthology.orchestra.run.trial097.stdout:x86_64
2026-02-23T17:57:30.181 DEBUG:teuthology.orchestra.run.trial097:> cat /etc/os-release
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:NAME="Rocky Linux"
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:ID="rocky"
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:VERSION_ID="10.1"
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:57:30.239 INFO:teuthology.orchestra.run.trial097.stdout:ANSI_COLOR="0;32"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:VENDOR_NAME="RESF"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:57:30.240 INFO:teuthology.orchestra.run.trial097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:30.240 INFO:teuthology.lock.ops:Updating trial097.front.sepia.ceph.com on lock server
2026-02-23T17:57:30.304 INFO:teuthology.lock.ops:Updating [trial155.front.sepia.ceph.com]: set os type and version on server
2026-02-23T17:57:30.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial155.front.sepia.ceph.com'
2026-02-23T17:57:30.305 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T17:57:30.367 DEBUG:teuthology.orchestra.run.trial155:> true
2026-02-23T17:57:30.549 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial155.front.sepia.ceph.com'
2026-02-23T17:57:30.549 DEBUG:teuthology.orchestra.run.trial155:> uname -m
2026-02-23T17:57:30.607 INFO:teuthology.orchestra.run.trial155.stdout:x86_64
2026-02-23T17:57:30.607 DEBUG:teuthology.orchestra.run.trial155:> cat /etc/os-release
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:NAME="Rocky Linux"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:VERSION="10.1 (Red Quartz)"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:ID="rocky"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:ID_LIKE="rhel centos fedora"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:VERSION_ID="10.1"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:PLATFORM_ID="platform:el10"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:PRETTY_NAME="Rocky Linux 10.1 (Red Quartz)"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:ANSI_COLOR="0;32"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:LOGO="fedora-logo-icon"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:CPE_NAME="cpe:/o:rocky:rocky:10::baseos"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:HOME_URL="https://rockylinux.org/"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:VENDOR_NAME="RESF"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:VENDOR_URL="https://resf.org/"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:BUG_REPORT_URL="https://bugs.rockylinux.org/"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:SUPPORT_END="2035-05-31"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:ROCKY_SUPPORT_PRODUCT="Rocky-Linux-10"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:ROCKY_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:REDHAT_SUPPORT_PRODUCT="Rocky Linux"
2026-02-23T17:57:30.665 INFO:teuthology.orchestra.run.trial155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="10.1"
2026-02-23T17:57:30.666 INFO:teuthology.lock.ops:Updating trial155.front.sepia.ceph.com on lock server
2026-02-23T17:57:30.734 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T17:57:30.830 INFO:teuthology.dispatcher.supervisor:Running job 66602
2026-02-23T17:57:30.830 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_nmordech@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/nmordech-2026-02-23_13:43:25-rados-wip-rocky10-branch-of-the-day-2026-02-23-1771833711-distro-default-trial/66602 --name nmordech-2026-02-23_13:43:25-rados-wip-rocky10-branch-of-the-day-2026-02-23-1771833711-distro-default-trial --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore/{alloc$/{stupid} base mem$/{normal-1} onode-segment$/{512K} write$/{v1/{compr$/{yes$/{lz4}} v1}}}} rados supported-random-distro$/{rpm_latest} thrashers/pggrow_host thrashosds-health workloads/snaps-few-objects-balanced} -- /home/teuthworker/mnt/teuthology/nmordech-2026-02-23_13:43:25-rados-wip-rocky10-branch-of-the-day-2026-02-23-1771833711-distro-default-trial/66602/orig.config.yaml
2026-02-23T17:57:30.831 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/nmordech-2026-02-23_13:43:25-rados-wip-rocky10-branch-of-the-day-2026-02-23-1771833711-distro-default-trial/66602
2026-02-23T17:57:30.831 INFO:teuthology.dispatcher.supervisor:Job PID: 963755
2026-02-23T17:57:30.831 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-23T17:59:30.930 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:01:31.104 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:03:31.225 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:05:31.364 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:07:31.523 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:09:31.671 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:11:31.840 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:13:31.991 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:15:32.175 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:17:32.382 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:19:32.751 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:21:32.954 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T18:21:33.054 INFO:teuthology.dispatcher.supervisor:Success!
2026-02-23T18:21:33.278 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2026-02-23T18:21:34.210 INFO:teuthology.orchestra.console.trial097:Power off
2026-02-23T18:21:34.210 DEBUG:teuthology.orchestra.console.trial097:pexpect command: ipmitool -H trial097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T18:21:34.268 INFO:teuthology.orchestra.console.trial078:Power off
2026-02-23T18:21:34.268 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T18:21:34.279 INFO:teuthology.orchestra.console.trial155:Power off
2026-02-23T18:21:34.279 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T18:21:34.284 DEBUG:teuthology.orchestra.console.trial097:power off output: Chassis Power Control: Down/Off
2026-02-23T18:21:34.289 INFO:teuthology.orchestra.console.trial041:Power off
2026-02-23T18:21:34.289 DEBUG:teuthology.orchestra.console.trial041:pexpect command: ipmitool -H trial041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T18:21:34.306 DEBUG:teuthology.orchestra.console.trial078:power off output: Chassis Power Control: Down/Off
2026-02-23T18:21:34.316 DEBUG:teuthology.orchestra.console.trial155:power off output: Chassis Power Control: Down/Off
2026-02-23T18:21:34.325 DEBUG:teuthology.orchestra.console.trial041:power off output: Chassis Power Control: Down/Off
2026-02-23T18:21:38.287 DEBUG:teuthology.orchestra.console.trial097:pexpect command: ipmitool -H trial097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:38.307 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:38.317 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:38.325 DEBUG:teuthology.orchestra.console.trial041:pexpect command: ipmitool -H trial041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:38.396 DEBUG:teuthology.orchestra.console.trial097:check power output: Chassis Power is on
2026-02-23T18:21:38.414 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is on
2026-02-23T18:21:38.426 DEBUG:teuthology.orchestra.console.trial155:check power output: Chassis Power is on
2026-02-23T18:21:38.433 DEBUG:teuthology.orchestra.console.trial041:check power output: Chassis Power is on
2026-02-23T18:21:42.401 DEBUG:teuthology.orchestra.console.trial097:pexpect command: ipmitool -H trial097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:42.415 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:42.427 DEBUG:teuthology.orchestra.console.trial155:pexpect command: ipmitool -H trial155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:42.434 DEBUG:teuthology.orchestra.console.trial041:pexpect command: ipmitool -H trial041.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T18:21:42.511 DEBUG:teuthology.orchestra.console.trial097:check power output: Chassis Power is off
2026-02-23T18:21:42.511 INFO:teuthology.orchestra.console.trial097:Power off completed
2026-02-23T18:21:42.524 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is off
2026-02-23T18:21:42.524 INFO:teuthology.orchestra.console.trial078:Power off completed
2026-02-23T18:21:42.536 DEBUG:teuthology.orchestra.console.trial155:check power output: Chassis Power is off
2026-02-23T18:21:42.536 INFO:teuthology.orchestra.console.trial155:Power off completed
2026-02-23T18:21:42.543 DEBUG:teuthology.orchestra.console.trial041:check power output: Chassis Power is off
2026-02-23T18:21:42.543 INFO:teuthology.orchestra.console.trial041:Power off completed
2026-02-23T18:21:42.855 INFO:teuthology.lock.ops:unlocked: trial097.front.sepia.ceph.com
2026-02-23T18:21:42.862 INFO:teuthology.lock.ops:unlocked: trial155.front.sepia.ceph.com
2026-02-23T18:21:42.866 INFO:teuthology.lock.ops:unlocked: trial078.front.sepia.ceph.com
2026-02-23T18:21:42.874 INFO:teuthology.lock.ops:unlocked: trial041.front.sepia.ceph.com