2026-02-23T04:47:03.253 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-23T04:47:03.253 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T04:47:03.330 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T04:47:03.584 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7ac376866120>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7ac376070040>, signals=[15])
2026-02-23T04:47:03.584 INFO:teuthology.lock.ops:Start node 'trial078.front.sepia.ceph.com' reimaging
2026-02-23T04:47:03.584 INFO:teuthology.lock.ops:Updating [trial078.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T04:47:03.584 INFO:teuthology.lock.ops:Updating trial078.front.sepia.ceph.com on lock server
2026-02-23T04:47:03.640 INFO:teuthology.lock.ops:Start node 'trial103.front.sepia.ceph.com' reimaging
2026-02-23T04:47:03.641 INFO:teuthology.lock.ops:Updating [trial103.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T04:47:03.641 INFO:teuthology.lock.ops:Updating trial103.front.sepia.ceph.com on lock server
2026-02-23T04:47:03.701 INFO:teuthology.lock.ops:Start node 'trial150.front.sepia.ceph.com' reimaging
2026-02-23T04:47:03.701 INFO:teuthology.lock.ops:Updating [trial150.front.sepia.ceph.com]: reset os type and version on server
2026-02-23T04:47:03.701 INFO:teuthology.lock.ops:Updating trial150.front.sepia.ceph.com on lock server
2026-02-23T04:47:03.943 DEBUG:teuthology.provision.fog.trial103:Requesting image trial_centos_9.stream (ID 3)
2026-02-23T04:47:03.944 DEBUG:teuthology.provision.fog.trial078:Requesting image trial_centos_9.stream (ID 3)
2026-02-23T04:47:03.997 INFO:teuthology.provision.fog.trial103:Scheduling deploy of centos 9.stream
2026-02-23T04:47:03.999 DEBUG:teuthology.provision.fog.trial150:Requesting image trial_centos_9.stream (ID 3)
2026-02-23T04:47:04.004 INFO:teuthology.provision.fog.trial078:Scheduling deploy of centos 9.stream
2026-02-23T04:47:04.048 INFO:teuthology.provision.fog.trial150:Scheduling deploy of centos 9.stream
2026-02-23T04:47:05.443 INFO:teuthology.orchestra.console.trial103:Power off
2026-02-23T04:47:05.443 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T04:47:05.481 DEBUG:teuthology.orchestra.console.trial103:power off output: Chassis Power Control: Down/Off
2026-02-23T04:47:05.590 INFO:teuthology.orchestra.console.trial078:Power off
2026-02-23T04:47:05.590 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T04:47:05.607 INFO:teuthology.orchestra.console.trial150:Power off
2026-02-23T04:47:05.608 DEBUG:teuthology.orchestra.console.trial150:pexpect command: ipmitool -H trial150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T04:47:05.629 DEBUG:teuthology.orchestra.console.trial078:power off output: Chassis Power Control: Down/Off
2026-02-23T04:47:05.645 DEBUG:teuthology.orchestra.console.trial150:power off output: Chassis Power Control: Down/Off
2026-02-23T04:47:09.486 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T04:47:09.594 DEBUG:teuthology.orchestra.console.trial103:check power output: Chassis Power is off
2026-02-23T04:47:09.594 INFO:teuthology.orchestra.console.trial103:Power off completed
2026-02-23T04:47:09.629 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T04:47:09.646 DEBUG:teuthology.orchestra.console.trial150:pexpect command: ipmitool -H trial150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T04:47:09.694 INFO:teuthology.orchestra.console.trial103:Power on
2026-02-23T04:47:09.695 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T04:47:09.737 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is off
2026-02-23T04:47:09.737 INFO:teuthology.orchestra.console.trial078:Power off completed
2026-02-23T04:47:09.739 DEBUG:teuthology.orchestra.console.trial103:power on output: Chassis Power Control: Up/On
2026-02-23T04:47:09.753 DEBUG:teuthology.orchestra.console.trial150:check power output: Chassis Power is off
2026-02-23T04:47:09.753 INFO:teuthology.orchestra.console.trial150:Power off completed
2026-02-23T04:47:09.838 INFO:teuthology.orchestra.console.trial078:Power on
2026-02-23T04:47:09.839 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T04:47:09.854 INFO:teuthology.orchestra.console.trial150:Power on
2026-02-23T04:47:09.854 DEBUG:teuthology.orchestra.console.trial150:pexpect command: ipmitool -H trial150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-23T04:47:09.879 DEBUG:teuthology.orchestra.console.trial078:power on output: Chassis Power Control: Up/On
2026-02-23T04:47:09.893 DEBUG:teuthology.orchestra.console.trial150:power on output: Chassis Power Control: Up/On
2026-02-23T04:47:13.743 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T04:47:13.854 DEBUG:teuthology.orchestra.console.trial103:check power output: Chassis Power is off
2026-02-23T04:47:13.879 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T04:47:13.894 DEBUG:teuthology.orchestra.console.trial150:pexpect command: ipmitool -H trial150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T04:47:13.989 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is on
2026-02-23T04:47:13.989 INFO:teuthology.orchestra.console.trial078:Power on completed
2026-02-23T04:47:14.002 DEBUG:teuthology.orchestra.console.trial150:check power output: Chassis Power is on
2026-02-23T04:47:14.002 INFO:teuthology.orchestra.console.trial150:Power on completed
2026-02-23T04:47:14.090 INFO:teuthology.provision.fog.trial078:Waiting for deploy to finish
2026-02-23T04:47:14.103 INFO:teuthology.provision.fog.trial150:Waiting for deploy to finish
2026-02-23T04:47:17.858 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T04:47:17.967 DEBUG:teuthology.orchestra.console.trial103:check power output: Chassis Power is on
2026-02-23T04:47:17.968 INFO:teuthology.orchestra.console.trial103:Power on completed
2026-02-23T04:47:18.068 INFO:teuthology.provision.fog.trial103:Waiting for deploy to finish
2026-02-23T04:49:02.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:02.453 ERROR:teuthology.orchestra.connection:Error authenticating with trial078.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2026-02-23T04:49:05.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:08.416 WARNING:teuthology.provision.fog.trial103:[Errno None] Unable to connect to port 22 on 10.20.193.103
2026-02-23T04:49:14.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:17.504 WARNING:teuthology.provision.fog.trial103:[Errno None] Unable to connect to port 22 on 10.20.193.103
2026-02-23T04:49:17.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:20.896 WARNING:teuthology.provision.fog.trial150:[Errno None] Unable to connect to port 22 on 10.20.193.150
2026-02-23T04:49:23.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:26.592 WARNING:teuthology.provision.fog.trial103:[Errno None] Unable to connect to port 22 on 10.20.193.103
2026-02-23T04:49:26.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:29.983 WARNING:teuthology.provision.fog.trial150:[Errno None] Unable to connect to port 22 on 10.20.193.150
2026-02-23T04:49:32.594 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:35.680 WARNING:teuthology.provision.fog.trial103:[Errno None] Unable to connect to port 22 on 10.20.193.103
2026-02-23T04:49:35.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:39.072 WARNING:teuthology.provision.fog.trial150:[Errno None] Unable to connect to port 22 on 10.20.193.150
2026-02-23T04:49:41.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:44.768 WARNING:teuthology.provision.fog.trial103:[Errno None] Unable to connect to port 22 on 10.20.193.103
2026-02-23T04:49:45.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:48.160 WARNING:teuthology.provision.fog.trial150:[Errno None] Unable to connect to port 22 on 10.20.193.150
2026-02-23T04:49:50.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:49:54.162 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:50:06.456 WARNING:teuthology.provision.fog.trial078:timed out
2026-02-23T04:50:12.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:50:15.666 ERROR:teuthology.orchestra.connection:Error authenticating with trial078.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T04:50:19.784 ERROR:teuthology.orchestra.connection:Error authenticating with trial078.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-23T04:50:26.941 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T04:50:26.942 DEBUG:teuthology.orchestra.run.trial078:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T04:50:26.961 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T04:50:26.961 DEBUG:teuthology.orchestra.run.trial103:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T04:50:27.078 INFO:teuthology.provision.fog.trial078:Node is ready
2026-02-23T04:50:27.079 DEBUG:teuthology.orchestra.run.trial078:> hostname
2026-02-23T04:50:27.117 INFO:teuthology.provision.fog.trial103:Node is ready
2026-02-23T04:50:27.118 DEBUG:teuthology.orchestra.run.trial103:> hostname
2026-02-23T04:50:27.135 INFO:teuthology.orchestra.run.trial078.stdout:trial078.front.sepia.ceph.com
2026-02-23T04:50:27.135 DEBUG:teuthology.orchestra.run.trial078:> grep trial078.front.sepia.ceph.com /etc/hosts
2026-02-23T04:50:27.172 INFO:teuthology.orchestra.run.trial103.stdout:trial103.front.sepia.ceph.com
2026-02-23T04:50:27.173 DEBUG:teuthology.orchestra.run.trial103:> grep trial103.front.sepia.ceph.com /etc/hosts
2026-02-23T04:50:27.193 INFO:teuthology.orchestra.run.trial078.stdout:10.20.193.78 trial078.front.sepia.ceph.com trial078
2026-02-23T04:50:27.193 DEBUG:teuthology.orchestra.run.trial078:> sudo hostname trial078
2026-02-23T04:50:27.230 INFO:teuthology.orchestra.run.trial103.stdout:10.20.193.103 trial103.front.sepia.ceph.com trial103
2026-02-23T04:50:27.230 DEBUG:teuthology.orchestra.run.trial103:> sudo hostname trial103
2026-02-23T04:50:27.263 DEBUG:teuthology.orchestra.run.trial078:> sudo sed -i -e 's/trial078.front.sepia.ceph.com/trial078/g' /etc/hosts
2026-02-23T04:50:27.299 DEBUG:teuthology.orchestra.run.trial103:> sudo sed -i -e 's/trial103.front.sepia.ceph.com/trial103/g' /etc/hosts
2026-02-23T04:50:27.328 DEBUG:teuthology.orchestra.run.trial078:> sudo sed -i -e 's/10.20.193.78/10.20.193.78/g' /etc/hosts
2026-02-23T04:50:27.365 DEBUG:teuthology.orchestra.run.trial103:> sudo sed -i -e 's/10.20.193.103/10.20.193.103/g' /etc/hosts
2026-02-23T04:50:27.395 DEBUG:teuthology.orchestra.run.trial078:> sudo sed -i -e 's/trial078.front.sepia.ceph.com/trial078/g' /etc/hostname
2026-02-23T04:50:27.430 DEBUG:teuthology.orchestra.run.trial103:> sudo sed -i -e 's/trial103.front.sepia.ceph.com/trial103/g' /etc/hostname
2026-02-23T04:50:27.461 DEBUG:teuthology.orchestra.run.trial078:> sudo hostname trial078
2026-02-23T04:50:27.495 DEBUG:teuthology.orchestra.run.trial103:> sudo hostname trial103
2026-02-23T04:50:27.528 DEBUG:teuthology.orchestra.run.trial078:> cat /etc/os-release
2026-02-23T04:50:27.559 DEBUG:teuthology.orchestra.run.trial103:> cat /etc/os-release
2026-02-23T04:50:27.584 INFO:teuthology.orchestra.run.trial078.stdout:NAME="CentOS Stream"
2026-02-23T04:50:27.584 INFO:teuthology.orchestra.run.trial078.stdout:VERSION="9"
2026-02-23T04:50:27.584 INFO:teuthology.orchestra.run.trial078.stdout:ID="centos"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:ID_LIKE="rhel fedora"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:VERSION_ID="9"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:PLATFORM_ID="platform:el9"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:ANSI_COLOR="0;31"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:LOGO="fedora-logo-icon"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:HOME_URL="https://centos.org/"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T04:50:27.585 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T04:50:27.587 INFO:teuthology.provision.fog.trial078:Deploy complete!
2026-02-23T04:50:27.588 DEBUG:teuthology.parallel:result is None
2026-02-23T04:50:27.615 INFO:teuthology.orchestra.run.trial103.stdout:NAME="CentOS Stream"
2026-02-23T04:50:27.615 INFO:teuthology.orchestra.run.trial103.stdout:VERSION="9"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:ID="centos"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:ID_LIKE="rhel fedora"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:VERSION_ID="9"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:PLATFORM_ID="platform:el9"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:ANSI_COLOR="0;31"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:LOGO="fedora-logo-icon"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:HOME_URL="https://centos.org/"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T04:50:27.616 INFO:teuthology.orchestra.run.trial103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T04:50:27.617 INFO:teuthology.orchestra.run.trial103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T04:50:27.617 INFO:teuthology.provision.fog.trial103:Deploy complete!
2026-02-23T04:50:27.618 DEBUG:teuthology.parallel:result is None
2026-02-23T04:50:30.013 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-23T04:50:30.013 DEBUG:teuthology.orchestra.run.trial150:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-23T04:50:30.168 INFO:teuthology.provision.fog.trial150:Node is ready
2026-02-23T04:50:30.169 DEBUG:teuthology.orchestra.run.trial150:> hostname
2026-02-23T04:50:30.225 INFO:teuthology.orchestra.run.trial150.stdout:trial150.front.sepia.ceph.com
2026-02-23T04:50:30.225 DEBUG:teuthology.orchestra.run.trial150:> grep trial150.front.sepia.ceph.com /etc/hosts
2026-02-23T04:50:30.282 INFO:teuthology.orchestra.run.trial150.stdout:10.20.193.150 trial150.front.sepia.ceph.com trial150
2026-02-23T04:50:30.282 DEBUG:teuthology.orchestra.run.trial150:> sudo hostname trial150
2026-02-23T04:50:30.351 DEBUG:teuthology.orchestra.run.trial150:> sudo sed -i -e 's/trial150.front.sepia.ceph.com/trial150/g' /etc/hosts
2026-02-23T04:50:30.416 DEBUG:teuthology.orchestra.run.trial150:> sudo sed -i -e 's/10.20.193.150/10.20.193.150/g' /etc/hosts
2026-02-23T04:50:30.480 DEBUG:teuthology.orchestra.run.trial150:> sudo sed -i -e 's/trial150.front.sepia.ceph.com/trial150/g' /etc/hostname
2026-02-23T04:50:30.545 DEBUG:teuthology.orchestra.run.trial150:> sudo hostname trial150
2026-02-23T04:50:30.609 DEBUG:teuthology.orchestra.run.trial150:> cat /etc/os-release
2026-02-23T04:50:30.666 INFO:teuthology.orchestra.run.trial150.stdout:NAME="CentOS Stream"
2026-02-23T04:50:30.666 INFO:teuthology.orchestra.run.trial150.stdout:VERSION="9"
2026-02-23T04:50:30.666 INFO:teuthology.orchestra.run.trial150.stdout:ID="centos"
2026-02-23T04:50:30.666 INFO:teuthology.orchestra.run.trial150.stdout:ID_LIKE="rhel fedora"
2026-02-23T04:50:30.666 INFO:teuthology.orchestra.run.trial150.stdout:VERSION_ID="9"
2026-02-23T04:50:30.666 INFO:teuthology.orchestra.run.trial150.stdout:PLATFORM_ID="platform:el9"
2026-02-23T04:50:30.666 INFO:teuthology.orchestra.run.trial150.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T04:50:30.667 INFO:teuthology.orchestra.run.trial150.stdout:ANSI_COLOR="0;31"
2026-02-23T04:50:30.667 INFO:teuthology.orchestra.run.trial150.stdout:LOGO="fedora-logo-icon"
2026-02-23T04:50:30.667 INFO:teuthology.orchestra.run.trial150.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T04:50:30.667 INFO:teuthology.orchestra.run.trial150.stdout:HOME_URL="https://centos.org/"
2026-02-23T04:50:30.667 INFO:teuthology.orchestra.run.trial150.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T04:50:30.667 INFO:teuthology.orchestra.run.trial150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T04:50:30.668 INFO:teuthology.orchestra.run.trial150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T04:50:30.668 INFO:teuthology.provision.fog.trial150:Deploy complete!
2026-02-23T04:50:30.669 DEBUG:teuthology.parallel:result is None
2026-02-23T04:50:31.199 INFO:teuthology.lock.ops:Checking trial078.front.sepia.ceph.com
2026-02-23T04:50:31.200 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T04:50:31.312 INFO:teuthology.lock.ops:Checking trial103.front.sepia.ceph.com
2026-02-23T04:50:31.313 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T04:50:31.377 INFO:teuthology.lock.ops:Checking trial150.front.sepia.ceph.com
2026-02-23T04:50:31.377 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-23T04:50:31.433 INFO:teuthology.lock.ops:Updating [trial078.front.sepia.ceph.com]: set os type and version on server
2026-02-23T04:50:31.434 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial078.front.sepia.ceph.com'
2026-02-23T04:50:31.434 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:50:31.496 DEBUG:teuthology.orchestra.run.trial078:> true
2026-02-23T04:50:31.573 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial078.front.sepia.ceph.com'
2026-02-23T04:50:31.573 DEBUG:teuthology.orchestra.run.trial078:> uname -m
2026-02-23T04:50:31.629 INFO:teuthology.orchestra.run.trial078.stdout:x86_64
2026-02-23T04:50:31.629 DEBUG:teuthology.orchestra.run.trial078:> cat /etc/os-release
2026-02-23T04:50:31.685 INFO:teuthology.orchestra.run.trial078.stdout:NAME="CentOS Stream"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:VERSION="9"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:ID="centos"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:ID_LIKE="rhel fedora"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:VERSION_ID="9"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:PLATFORM_ID="platform:el9"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:ANSI_COLOR="0;31"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:LOGO="fedora-logo-icon"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T04:50:31.686 INFO:teuthology.orchestra.run.trial078.stdout:HOME_URL="https://centos.org/"
2026-02-23T04:50:31.687 INFO:teuthology.orchestra.run.trial078.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T04:50:31.687 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T04:50:31.687 INFO:teuthology.orchestra.run.trial078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T04:50:31.687 INFO:teuthology.lock.ops:Updating trial078.front.sepia.ceph.com on lock server
2026-02-23T04:50:31.793 INFO:teuthology.lock.ops:Updating [trial103.front.sepia.ceph.com]: set os type and version on server
2026-02-23T04:50:31.793 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial103.front.sepia.ceph.com'
2026-02-23T04:50:31.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:50:31.863 DEBUG:teuthology.orchestra.run.trial103:> true
2026-02-23T04:50:31.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial103.front.sepia.ceph.com'
2026-02-23T04:50:31.933 DEBUG:teuthology.orchestra.run.trial103:> uname -m
2026-02-23T04:50:31.989 INFO:teuthology.orchestra.run.trial103.stdout:x86_64
2026-02-23T04:50:31.989 DEBUG:teuthology.orchestra.run.trial103:> cat /etc/os-release
2026-02-23T04:50:32.046 INFO:teuthology.orchestra.run.trial103.stdout:NAME="CentOS Stream"
2026-02-23T04:50:32.046 INFO:teuthology.orchestra.run.trial103.stdout:VERSION="9"
2026-02-23T04:50:32.046 INFO:teuthology.orchestra.run.trial103.stdout:ID="centos"
2026-02-23T04:50:32.046 INFO:teuthology.orchestra.run.trial103.stdout:ID_LIKE="rhel fedora"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:VERSION_ID="9"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:PLATFORM_ID="platform:el9"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:ANSI_COLOR="0;31"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:LOGO="fedora-logo-icon"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:HOME_URL="https://centos.org/"
2026-02-23T04:50:32.047 INFO:teuthology.orchestra.run.trial103.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T04:50:32.048 INFO:teuthology.orchestra.run.trial103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T04:50:32.048 INFO:teuthology.orchestra.run.trial103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T04:50:32.048 INFO:teuthology.lock.ops:Updating trial103.front.sepia.ceph.com on lock server
2026-02-23T04:50:32.150 INFO:teuthology.lock.ops:Updating [trial150.front.sepia.ceph.com]: set os type and version on server
2026-02-23T04:50:32.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial150.front.sepia.ceph.com'
2026-02-23T04:50:32.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-23T04:50:32.221 DEBUG:teuthology.orchestra.run.trial150:> true
2026-02-23T04:50:32.294 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial150.front.sepia.ceph.com'
2026-02-23T04:50:32.294 DEBUG:teuthology.orchestra.run.trial150:> uname -m
2026-02-23T04:50:32.350 INFO:teuthology.orchestra.run.trial150.stdout:x86_64
2026-02-23T04:50:32.351 DEBUG:teuthology.orchestra.run.trial150:> cat /etc/os-release
2026-02-23T04:50:32.406 INFO:teuthology.orchestra.run.trial150.stdout:NAME="CentOS Stream"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:VERSION="9"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:ID="centos"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:ID_LIKE="rhel fedora"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:VERSION_ID="9"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:PLATFORM_ID="platform:el9"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:ANSI_COLOR="0;31"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:LOGO="fedora-logo-icon"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:HOME_URL="https://centos.org/"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-23T04:50:32.407 INFO:teuthology.orchestra.run.trial150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-23T04:50:32.408 INFO:teuthology.lock.ops:Updating trial150.front.sepia.ceph.com on lock server
2026-02-23T04:50:32.600 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T04:50:32.700 INFO:teuthology.dispatcher.supervisor:Running job 64847
2026-02-23T04:50:32.700 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_vshankar@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/vshankar-2026-02-23_04:04:18-fs-wip-vshankar-testing-20260222.101816-testing-default-trial/64847 --name vshankar-2026-02-23_04:04:18-fs-wip-vshankar-testing-20260222.101816-testing-default-trial --description fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} -- /home/teuthworker/mnt/teuthology/vshankar-2026-02-23_04:04:18-fs-wip-vshankar-testing-20260222.101816-testing-default-trial/64847/orig.config.yaml
2026-02-23T04:50:32.701 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/vshankar-2026-02-23_04:04:18-fs-wip-vshankar-testing-20260222.101816-testing-default-trial/64847
2026-02-23T04:50:32.701 INFO:teuthology.dispatcher.supervisor:Job PID: 1700866
2026-02-23T04:50:32.701 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-23T04:52:32.702 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T04:54:32.824 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T04:56:32.945 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T04:58:33.035 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T05:00:33.160 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T05:02:33.238 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T05:04:33.358 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T05:06:33.475 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T05:08:33.601 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T05:10:33.723 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-23T05:10:33.850 INFO:teuthology.dispatcher.supervisor:Success!
2026-02-23T05:10:34.030 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2026-02-23T05:10:34.781 INFO:teuthology.orchestra.console.trial078:Power off
2026-02-23T05:10:34.781 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T05:10:34.789 INFO:teuthology.orchestra.console.trial103:Power off
2026-02-23T05:10:34.790 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T05:10:34.812 INFO:teuthology.orchestra.console.trial150:Power off
2026-02-23T05:10:34.812 DEBUG:teuthology.orchestra.console.trial150:pexpect command: ipmitool -H trial150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-23T05:10:34.820 DEBUG:teuthology.orchestra.console.trial078:power off output: Chassis Power Control: Down/Off
2026-02-23T05:10:34.828 DEBUG:teuthology.orchestra.console.trial103:power off output: Chassis Power Control: Down/Off
2026-02-23T05:10:34.854 DEBUG:teuthology.orchestra.console.trial150:power off output: Chassis Power Control: Down/Off
2026-02-23T05:10:38.821 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T05:10:38.829 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T05:10:38.854 DEBUG:teuthology.orchestra.console.trial150:pexpect command: ipmitool -H trial150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T05:10:38.930 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is on
2026-02-23T05:10:38.936 DEBUG:teuthology.orchestra.console.trial103:check power output: Chassis Power is on
2026-02-23T05:10:38.964 DEBUG:teuthology.orchestra.console.trial150:check power output: Chassis Power is on
2026-02-23T05:10:42.931 DEBUG:teuthology.orchestra.console.trial078:pexpect command: ipmitool -H trial078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T05:10:42.937 DEBUG:teuthology.orchestra.console.trial103:pexpect command: ipmitool -H trial103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T05:10:42.964 DEBUG:teuthology.orchestra.console.trial150:pexpect command: ipmitool -H trial150.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-23T05:10:43.039 DEBUG:teuthology.orchestra.console.trial078:check power output: Chassis Power is off
2026-02-23T05:10:43.039 INFO:teuthology.orchestra.console.trial078:Power off completed
2026-02-23T05:10:43.044 DEBUG:teuthology.orchestra.console.trial103:check power output: Chassis Power is off
2026-02-23T05:10:43.044 INFO:teuthology.orchestra.console.trial103:Power off completed
2026-02-23T05:10:43.071 DEBUG:teuthology.orchestra.console.trial150:check power output: Chassis Power is off
2026-02-23T05:10:43.071 INFO:teuthology.orchestra.console.trial150:Power off completed
2026-02-23T05:10:43.333 INFO:teuthology.lock.ops:unlocked: trial103.front.sepia.ceph.com
2026-02-23T05:10:43.333 INFO:teuthology.lock.ops:unlocked: trial078.front.sepia.ceph.com
2026-02-23T05:10:43.346 INFO:teuthology.lock.ops:unlocked: trial150.front.sepia.ceph.com