2026-02-14T01:43:21.220 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71
2026-02-14T01:43:21.220 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:43:21.296 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:43:21.559 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7abf08cce6c0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7abf08489940>, signals=[15])
2026-02-14T01:43:21.559 INFO:teuthology.lock.ops:Start node 'trial028.front.sepia.ceph.com' reimaging
2026-02-14T01:43:21.559 INFO:teuthology.lock.ops:Updating [trial028.front.sepia.ceph.com]: reset os type and version on server
2026-02-14T01:43:21.559 INFO:teuthology.lock.ops:Updating trial028.front.sepia.ceph.com on lock server
2026-02-14T01:43:21.618 INFO:teuthology.lock.ops:Start node 'trial148.front.sepia.ceph.com' reimaging
2026-02-14T01:43:21.618 INFO:teuthology.lock.ops:Updating [trial148.front.sepia.ceph.com]: reset os type and version on server
2026-02-14T01:43:21.618 INFO:teuthology.lock.ops:Updating trial148.front.sepia.ceph.com on lock server
2026-02-14T01:43:21.676 INFO:teuthology.lock.ops:Start node 'trial153.front.sepia.ceph.com' reimaging
2026-02-14T01:43:21.676 INFO:teuthology.lock.ops:Updating [trial153.front.sepia.ceph.com]: reset os type and version on server
2026-02-14T01:43:21.677 INFO:teuthology.lock.ops:Updating trial153.front.sepia.ceph.com on lock server
2026-02-14T01:43:21.915 DEBUG:teuthology.provision.fog.trial148:Requesting image trial_centos_9.stream (ID 3)
2026-02-14T01:43:21.916 DEBUG:teuthology.provision.fog.trial028:Requesting image trial_centos_9.stream (ID 3)
2026-02-14T01:43:21.959 INFO:teuthology.provision.fog.trial148:Scheduling deploy of centos 9.stream
2026-02-14T01:43:21.961 INFO:teuthology.provision.fog.trial028:Scheduling deploy of centos 9.stream
2026-02-14T01:43:21.971 DEBUG:teuthology.provision.fog.trial153:Requesting image trial_centos_9.stream (ID 3)
2026-02-14T01:43:22.025 INFO:teuthology.provision.fog.trial153:Scheduling deploy of centos 9.stream
2026-02-14T01:43:23.282 INFO:teuthology.orchestra.console.trial148:Power off
2026-02-14T01:43:23.283 DEBUG:teuthology.orchestra.console.trial148:pexpect command: ipmitool -H trial148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-14T01:43:23.301 INFO:teuthology.orchestra.console.trial028:Power off
2026-02-14T01:43:23.301 DEBUG:teuthology.orchestra.console.trial028:pexpect command: ipmitool -H trial028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-14T01:43:23.312 INFO:teuthology.orchestra.console.trial153:Power off
2026-02-14T01:43:23.312 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-14T01:43:23.322 DEBUG:teuthology.orchestra.console.trial148:power off output: Chassis Power Control: Down/Off
2026-02-14T01:43:23.340 DEBUG:teuthology.orchestra.console.trial028:power off output: Chassis Power Control: Down/Off
2026-02-14T01:43:23.351 DEBUG:teuthology.orchestra.console.trial153:power off output: Chassis Power Control: Down/Off
2026-02-14T01:43:27.326 DEBUG:teuthology.orchestra.console.trial148:pexpect command: ipmitool -H trial148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T01:43:27.340 DEBUG:teuthology.orchestra.console.trial028:pexpect command: ipmitool -H trial028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T01:43:27.352 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T01:43:27.434 DEBUG:teuthology.orchestra.console.trial148:check power output: Chassis Power is off
2026-02-14T01:43:27.435 INFO:teuthology.orchestra.console.trial148:Power off completed
2026-02-14T01:43:27.448 DEBUG:teuthology.orchestra.console.trial028:check power output: Chassis Power is off
2026-02-14T01:43:27.448 INFO:teuthology.orchestra.console.trial028:Power off completed
2026-02-14T01:43:27.460 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is off
2026-02-14T01:43:27.460 INFO:teuthology.orchestra.console.trial153:Power off completed
2026-02-14T01:43:27.535 INFO:teuthology.orchestra.console.trial148:Power on
2026-02-14T01:43:27.536 DEBUG:teuthology.orchestra.console.trial148:pexpect command: ipmitool -H trial148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-14T01:43:27.549 INFO:teuthology.orchestra.console.trial028:Power on
2026-02-14T01:43:27.550 DEBUG:teuthology.orchestra.console.trial028:pexpect command: ipmitool -H trial028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-14T01:43:27.561 INFO:teuthology.orchestra.console.trial153:Power on
2026-02-14T01:43:27.561 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-14T01:43:27.575 DEBUG:teuthology.orchestra.console.trial148:power on output: Chassis Power Control: Up/On
2026-02-14T01:43:27.590 DEBUG:teuthology.orchestra.console.trial028:power on output: Chassis Power Control: Up/On
2026-02-14T01:43:27.600 DEBUG:teuthology.orchestra.console.trial153:power on output: Chassis Power Control: Up/On
2026-02-14T01:43:31.580 DEBUG:teuthology.orchestra.console.trial148:pexpect command: ipmitool -H trial148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T01:43:31.591 DEBUG:teuthology.orchestra.console.trial028:pexpect command: ipmitool -H trial028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T01:43:31.601 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T01:43:31.689 DEBUG:teuthology.orchestra.console.trial148:check power output: Chassis Power is on
2026-02-14T01:43:31.689 INFO:teuthology.orchestra.console.trial148:Power on completed
2026-02-14T01:43:31.701 DEBUG:teuthology.orchestra.console.trial028:check power output: Chassis Power is on
2026-02-14T01:43:31.701 INFO:teuthology.orchestra.console.trial028:Power on completed
2026-02-14T01:43:31.711 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is on
2026-02-14T01:43:31.711 INFO:teuthology.orchestra.console.trial153:Power on completed
2026-02-14T01:43:31.791 INFO:teuthology.provision.fog.trial148:Waiting for deploy to finish
2026-02-14T01:43:31.802 INFO:teuthology.provision.fog.trial028:Waiting for deploy to finish
2026-02-14T01:43:31.812 INFO:teuthology.provision.fog.trial153:Waiting for deploy to finish
2026-02-14T01:45:19.225 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:45:19.368 ERROR:teuthology.orchestra.connection:Error authenticating with trial028.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2026-02-14T01:45:19.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:45:19.498 ERROR:teuthology.orchestra.connection:Error authenticating with trial148.front.sepia.ceph.com: AuthenticationException('Authentication failed.')
2026-02-14T01:45:34.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:45:37.821 WARNING:teuthology.provision.fog.trial153:[Errno None] Unable to connect to port 22 on 10.20.193.153
2026-02-14T01:45:43.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:45:46.909 WARNING:teuthology.provision.fog.trial153:[Errno None] Unable to connect to port 22 on 10.20.193.153
2026-02-14T01:45:52.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:45:55.998 WARNING:teuthology.provision.fog.trial153:[Errno None] Unable to connect to port 22 on 10.20.193.153
2026-02-14T01:46:02.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:46:05.085 WARNING:teuthology.provision.fog.trial153:[Errno None] Unable to connect to port 22 on 10.20.193.153
2026-02-14T01:46:11.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:46:23.371 WARNING:teuthology.provision.fog.trial028:timed out
2026-02-14T01:46:23.501 WARNING:teuthology.provision.fog.trial148:timed out
2026-02-14T01:46:29.372 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:46:29.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:46:32.737 ERROR:teuthology.orchestra.connection:Error authenticating with trial148.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-14T01:46:34.659 ERROR:teuthology.orchestra.connection:Error authenticating with trial028.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-14T01:46:36.829 ERROR:teuthology.orchestra.connection:Error authenticating with trial148.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-14T01:46:38.748 ERROR:teuthology.orchestra.connection:Error authenticating with trial028.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-14T01:46:43.939 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-14T01:46:43.939 DEBUG:teuthology.orchestra.run.trial148:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-14T01:46:44.129 INFO:teuthology.provision.fog.trial148:Node is ready
2026-02-14T01:46:44.129 DEBUG:teuthology.orchestra.run.trial148:> hostname
2026-02-14T01:46:44.185 INFO:teuthology.orchestra.run.trial148.stdout:trial148.front.sepia.ceph.com
2026-02-14T01:46:44.186 DEBUG:teuthology.orchestra.run.trial148:> grep trial148.front.sepia.ceph.com /etc/hosts
2026-02-14T01:46:44.243 INFO:teuthology.orchestra.run.trial148.stdout:10.20.193.148 trial148.front.sepia.ceph.com trial148
2026-02-14T01:46:44.243 DEBUG:teuthology.orchestra.run.trial148:> sudo hostname trial148
2026-02-14T01:46:44.314 DEBUG:teuthology.orchestra.run.trial148:> sudo sed -i -e 's/trial148.front.sepia.ceph.com/trial148/g' /etc/hosts
2026-02-14T01:46:44.381 DEBUG:teuthology.orchestra.run.trial148:> sudo sed -i -e 's/10.20.193.148/10.20.193.148/g' /etc/hosts
2026-02-14T01:46:44.445 DEBUG:teuthology.orchestra.run.trial148:> sudo sed -i -e 's/trial148.front.sepia.ceph.com/trial148/g' /etc/hostname
2026-02-14T01:46:44.510 DEBUG:teuthology.orchestra.run.trial148:> sudo hostname trial148
2026-02-14T01:46:44.575 DEBUG:teuthology.orchestra.run.trial148:> cat /etc/os-release
2026-02-14T01:46:44.632 INFO:teuthology.orchestra.run.trial148.stdout:NAME="CentOS Stream"
2026-02-14T01:46:44.633 INFO:teuthology.orchestra.run.trial148.stdout:VERSION="9"
2026-02-14T01:46:44.633 INFO:teuthology.orchestra.run.trial148.stdout:ID="centos"
2026-02-14T01:46:44.633 INFO:teuthology.orchestra.run.trial148.stdout:ID_LIKE="rhel fedora"
2026-02-14T01:46:44.633 INFO:teuthology.orchestra.run.trial148.stdout:VERSION_ID="9"
2026-02-14T01:46:44.633 INFO:teuthology.orchestra.run.trial148.stdout:PLATFORM_ID="platform:el9"
2026-02-14T01:46:44.633 INFO:teuthology.orchestra.run.trial148.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-14T01:46:44.633 INFO:teuthology.orchestra.run.trial148.stdout:ANSI_COLOR="0;31"
2026-02-14T01:46:44.634 INFO:teuthology.orchestra.run.trial148.stdout:LOGO="fedora-logo-icon"
2026-02-14T01:46:44.634 INFO:teuthology.orchestra.run.trial148.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-14T01:46:44.634 INFO:teuthology.orchestra.run.trial148.stdout:HOME_URL="https://centos.org/"
2026-02-14T01:46:44.634 INFO:teuthology.orchestra.run.trial148.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-14T01:46:44.634 INFO:teuthology.orchestra.run.trial148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-14T01:46:44.634 INFO:teuthology.orchestra.run.trial148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-14T01:46:44.635 INFO:teuthology.provision.fog.trial148:Deploy complete!
2026-02-14T01:46:44.636 DEBUG:teuthology.parallel:result is None
2026-02-14T01:46:45.849 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-14T01:46:45.849 DEBUG:teuthology.orchestra.run.trial028:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-14T01:46:46.009 INFO:teuthology.provision.fog.trial028:Node is ready
2026-02-14T01:46:46.009 DEBUG:teuthology.orchestra.run.trial028:> hostname
2026-02-14T01:46:46.065 INFO:teuthology.orchestra.run.trial028.stdout:trial028.front.sepia.ceph.com
2026-02-14T01:46:46.066 DEBUG:teuthology.orchestra.run.trial028:> grep trial028.front.sepia.ceph.com /etc/hosts
2026-02-14T01:46:46.125 INFO:teuthology.orchestra.run.trial028.stdout:10.20.193.28 trial028.front.sepia.ceph.com trial028
2026-02-14T01:46:46.126 DEBUG:teuthology.orchestra.run.trial028:> sudo hostname trial028
2026-02-14T01:46:46.195 DEBUG:teuthology.orchestra.run.trial028:> sudo sed -i -e 's/trial028.front.sepia.ceph.com/trial028/g' /etc/hosts
2026-02-14T01:46:46.261 DEBUG:teuthology.orchestra.run.trial028:> sudo sed -i -e 's/10.20.193.28/10.20.193.28/g' /etc/hosts
2026-02-14T01:46:46.325 DEBUG:teuthology.orchestra.run.trial028:> sudo sed -i -e 's/trial028.front.sepia.ceph.com/trial028/g' /etc/hostname
2026-02-14T01:46:46.390 DEBUG:teuthology.orchestra.run.trial028:> sudo hostname trial028
2026-02-14T01:46:46.453 DEBUG:teuthology.orchestra.run.trial028:> cat /etc/os-release
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:NAME="CentOS Stream"
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:VERSION="9"
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:ID="centos"
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:ID_LIKE="rhel fedora"
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:VERSION_ID="9"
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:PLATFORM_ID="platform:el9"
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-14T01:46:46.510 INFO:teuthology.orchestra.run.trial028.stdout:ANSI_COLOR="0;31"
2026-02-14T01:46:46.511 INFO:teuthology.orchestra.run.trial028.stdout:LOGO="fedora-logo-icon"
2026-02-14T01:46:46.511 INFO:teuthology.orchestra.run.trial028.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-14T01:46:46.511 INFO:teuthology.orchestra.run.trial028.stdout:HOME_URL="https://centos.org/"
2026-02-14T01:46:46.511 INFO:teuthology.orchestra.run.trial028.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-14T01:46:46.511 INFO:teuthology.orchestra.run.trial028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-14T01:46:46.511 INFO:teuthology.orchestra.run.trial028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-14T01:46:46.512 INFO:teuthology.provision.fog.trial028:Deploy complete!
2026-02-14T01:46:46.512 DEBUG:teuthology.parallel:result is None
2026-02-14T01:46:46.916 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-14T01:46:46.917 DEBUG:teuthology.orchestra.run.trial153:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-14T01:46:47.076 INFO:teuthology.provision.fog.trial153:Node is ready
2026-02-14T01:46:47.076 DEBUG:teuthology.orchestra.run.trial153:> hostname
2026-02-14T01:46:47.132 INFO:teuthology.orchestra.run.trial153.stdout:trial153.front.sepia.ceph.com
2026-02-14T01:46:47.133 DEBUG:teuthology.orchestra.run.trial153:> grep trial153.front.sepia.ceph.com /etc/hosts
2026-02-14T01:46:47.189 INFO:teuthology.orchestra.run.trial153.stdout:10.20.193.153 trial153.front.sepia.ceph.com trial153
2026-02-14T01:46:47.190 DEBUG:teuthology.orchestra.run.trial153:> sudo hostname trial153
2026-02-14T01:46:47.258 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i -e 's/trial153.front.sepia.ceph.com/trial153/g' /etc/hosts
2026-02-14T01:46:47.322 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i -e 's/10.20.193.153/10.20.193.153/g' /etc/hosts
2026-02-14T01:46:47.387 DEBUG:teuthology.orchestra.run.trial153:> sudo sed -i -e 's/trial153.front.sepia.ceph.com/trial153/g' /etc/hostname
2026-02-14T01:46:47.454 DEBUG:teuthology.orchestra.run.trial153:> sudo hostname trial153
2026-02-14T01:46:47.518 DEBUG:teuthology.orchestra.run.trial153:> cat /etc/os-release
2026-02-14T01:46:47.574 INFO:teuthology.orchestra.run.trial153.stdout:NAME="CentOS Stream"
2026-02-14T01:46:47.574 INFO:teuthology.orchestra.run.trial153.stdout:VERSION="9"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:ID="centos"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:ID_LIKE="rhel fedora"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:VERSION_ID="9"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:PLATFORM_ID="platform:el9"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:ANSI_COLOR="0;31"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:LOGO="fedora-logo-icon"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:HOME_URL="https://centos.org/"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-14T01:46:47.575 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-14T01:46:47.576 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-14T01:46:47.576 INFO:teuthology.provision.fog.trial153:Deploy complete!
2026-02-14T01:46:47.577 DEBUG:teuthology.parallel:result is None
2026-02-14T01:46:48.118 INFO:teuthology.lock.ops:Checking trial028.front.sepia.ceph.com
2026-02-14T01:46:48.118 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-14T01:46:48.232 INFO:teuthology.lock.ops:Checking trial148.front.sepia.ceph.com
2026-02-14T01:46:48.232 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-14T01:46:48.303 INFO:teuthology.lock.ops:Checking trial153.front.sepia.ceph.com
2026-02-14T01:46:48.303 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-14T01:46:48.366 INFO:teuthology.lock.ops:Updating [trial028.front.sepia.ceph.com]: set os type and version on server
2026-02-14T01:46:48.366 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial028.front.sepia.ceph.com'
2026-02-14T01:46:48.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:46:48.440 DEBUG:teuthology.orchestra.run.trial028:> true
2026-02-14T01:46:48.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial028.front.sepia.ceph.com'
2026-02-14T01:46:48.518 DEBUG:teuthology.orchestra.run.trial028:> uname -m
2026-02-14T01:46:48.576 INFO:teuthology.orchestra.run.trial028.stdout:x86_64
2026-02-14T01:46:48.577 DEBUG:teuthology.orchestra.run.trial028:> cat /etc/os-release
2026-02-14T01:46:48.632 INFO:teuthology.orchestra.run.trial028.stdout:NAME="CentOS Stream"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:VERSION="9"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:ID="centos"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:ID_LIKE="rhel fedora"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:VERSION_ID="9"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:PLATFORM_ID="platform:el9"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:ANSI_COLOR="0;31"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:LOGO="fedora-logo-icon"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:HOME_URL="https://centos.org/"
2026-02-14T01:46:48.633 INFO:teuthology.orchestra.run.trial028.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-14T01:46:48.634 INFO:teuthology.orchestra.run.trial028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-14T01:46:48.634 INFO:teuthology.orchestra.run.trial028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-14T01:46:48.635 INFO:teuthology.lock.ops:Updating trial028.front.sepia.ceph.com on lock server
2026-02-14T01:46:48.768 INFO:teuthology.lock.ops:Updating [trial148.front.sepia.ceph.com]: set os type and version on server
2026-02-14T01:46:48.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial148.front.sepia.ceph.com'
2026-02-14T01:46:48.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:46:48.827 DEBUG:teuthology.orchestra.run.trial148:> true
2026-02-14T01:46:48.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial148.front.sepia.ceph.com'
2026-02-14T01:46:48.899 DEBUG:teuthology.orchestra.run.trial148:> uname -m
2026-02-14T01:46:48.956 INFO:teuthology.orchestra.run.trial148.stdout:x86_64
2026-02-14T01:46:48.956 DEBUG:teuthology.orchestra.run.trial148:> cat /etc/os-release
2026-02-14T01:46:49.013 INFO:teuthology.orchestra.run.trial148.stdout:NAME="CentOS Stream"
2026-02-14T01:46:49.013 INFO:teuthology.orchestra.run.trial148.stdout:VERSION="9"
2026-02-14T01:46:49.013 INFO:teuthology.orchestra.run.trial148.stdout:ID="centos"
2026-02-14T01:46:49.013 INFO:teuthology.orchestra.run.trial148.stdout:ID_LIKE="rhel fedora"
2026-02-14T01:46:49.013 INFO:teuthology.orchestra.run.trial148.stdout:VERSION_ID="9"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:PLATFORM_ID="platform:el9"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:ANSI_COLOR="0;31"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:LOGO="fedora-logo-icon"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:HOME_URL="https://centos.org/"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-14T01:46:49.014 INFO:teuthology.orchestra.run.trial148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-14T01:46:49.015 INFO:teuthology.lock.ops:Updating trial148.front.sepia.ceph.com on lock server
2026-02-14T01:46:49.124 INFO:teuthology.lock.ops:Updating [trial153.front.sepia.ceph.com]: set os type and version on server
2026-02-14T01:46:49.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial153.front.sepia.ceph.com'
2026-02-14T01:46:49.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-14T01:46:49.184 DEBUG:teuthology.orchestra.run.trial153:> true
2026-02-14T01:46:49.258 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial153.front.sepia.ceph.com'
2026-02-14T01:46:49.258 DEBUG:teuthology.orchestra.run.trial153:> uname -m
2026-02-14T01:46:49.323 INFO:teuthology.orchestra.run.trial153.stdout:x86_64
2026-02-14T01:46:49.323 DEBUG:teuthology.orchestra.run.trial153:> cat /etc/os-release
2026-02-14T01:46:49.401 INFO:teuthology.orchestra.run.trial153.stdout:NAME="CentOS Stream"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:VERSION="9"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:ID="centos"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:ID_LIKE="rhel fedora"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:VERSION_ID="9"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:PLATFORM_ID="platform:el9"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:ANSI_COLOR="0;31"
2026-02-14T01:46:49.402 INFO:teuthology.orchestra.run.trial153.stdout:LOGO="fedora-logo-icon"
2026-02-14T01:46:49.403 INFO:teuthology.orchestra.run.trial153.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-14T01:46:49.403 INFO:teuthology.orchestra.run.trial153.stdout:HOME_URL="https://centos.org/"
2026-02-14T01:46:49.403 INFO:teuthology.orchestra.run.trial153.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-14T01:46:49.403 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-14T01:46:49.403 INFO:teuthology.orchestra.run.trial153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-14T01:46:49.404 INFO:teuthology.lock.ops:Updating trial153.front.sepia.ceph.com on lock server
2026-02-14T01:46:49.503 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:46:49.596 INFO:teuthology.dispatcher.supervisor:Running job 50583
2026-02-14T01:46:49.596 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/bin/teuthology -v --owner scheduled_jcollin@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/jcollin-2026-02-14_00:12:18-fs-wip-jcollin-testing-20260213.113451-tentacle-distro-default-trial/50583 --name jcollin-2026-02-14_00:12:18-fs-wip-jcollin-testing-20260213.113451-tentacle-distro-default-trial --description fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} -- /home/teuthworker/mnt/teuthology/jcollin-2026-02-14_00:12:18-fs-wip-jcollin-testing-20260213.113451-tentacle-distro-default-trial/50583/orig.config.yaml
2026-02-14T01:46:49.596 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/jcollin-2026-02-14_00:12:18-fs-wip-jcollin-testing-20260213.113451-tentacle-distro-default-trial/50583
2026-02-14T01:46:49.596 INFO:teuthology.dispatcher.supervisor:Job PID: 1643993
2026-02-14T01:46:49.596 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-14T01:48:49.597 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:50:49.717 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:52:49.832 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:54:49.917 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:56:50.039 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T01:58:50.202 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T02:00:50.324 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T02:02:50.439 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-14T02:02:50.559 INFO:teuthology.dispatcher.supervisor:Success!
2026-02-14T02:02:50.754 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2026-02-14T02:02:51.498 INFO:teuthology.orchestra.console.trial148:Power off
2026-02-14T02:02:51.498 DEBUG:teuthology.orchestra.console.trial148:pexpect command: ipmitool -H trial148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-14T02:02:51.511 INFO:teuthology.orchestra.console.trial028:Power off
2026-02-14T02:02:51.511 DEBUG:teuthology.orchestra.console.trial028:pexpect command: ipmitool -H trial028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-14T02:02:51.522 INFO:teuthology.orchestra.console.trial153:Power off
2026-02-14T02:02:51.522 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-14T02:02:51.538 DEBUG:teuthology.orchestra.console.trial148:power off output: Chassis Power Control: Down/Off
2026-02-14T02:02:51.550 DEBUG:teuthology.orchestra.console.trial028:power off output: Chassis Power Control: Down/Off
2026-02-14T02:02:51.561 DEBUG:teuthology.orchestra.console.trial153:power off output: Chassis Power Control: Down/Off
2026-02-14T02:02:55.538 DEBUG:teuthology.orchestra.console.trial148:pexpect command: ipmitool -H trial148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T02:02:55.551 DEBUG:teuthology.orchestra.console.trial028:pexpect command: ipmitool -H trial028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T02:02:55.562 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T02:02:55.649 DEBUG:teuthology.orchestra.console.trial148:check power output: Chassis Power is on
2026-02-14T02:02:55.661 DEBUG:teuthology.orchestra.console.trial028:check power output: Chassis Power is on
2026-02-14T02:02:55.672 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is on
2026-02-14T02:02:59.650 DEBUG:teuthology.orchestra.console.trial148:pexpect command: ipmitool -H trial148.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T02:02:59.662 DEBUG:teuthology.orchestra.console.trial028:pexpect command: ipmitool -H trial028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T02:02:59.672 DEBUG:teuthology.orchestra.console.trial153:pexpect command: ipmitool -H trial153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-14T02:02:59.760 DEBUG:teuthology.orchestra.console.trial148:check power output: Chassis Power is off
2026-02-14T02:02:59.760 INFO:teuthology.orchestra.console.trial148:Power off completed
2026-02-14T02:02:59.772 DEBUG:teuthology.orchestra.console.trial028:check power output: Chassis Power is off
2026-02-14T02:02:59.772 INFO:teuthology.orchestra.console.trial028:Power off completed
2026-02-14T02:02:59.782 DEBUG:teuthology.orchestra.console.trial153:check power output: Chassis Power is off
2026-02-14T02:02:59.782 INFO:teuthology.orchestra.console.trial153:Power off completed
2026-02-14T02:03:00.057 INFO:teuthology.lock.ops:unlocked: trial148.front.sepia.ceph.com
2026-02-14T02:03:00.067 INFO:teuthology.lock.ops:unlocked: trial028.front.sepia.ceph.com
2026-02-14T02:03:00.075 INFO:teuthology.lock.ops:unlocked: trial153.front.sepia.ceph.com