2026-02-24T15:44:57.327 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T15:44:57.327 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:44:57.419 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:44:57.549 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x74af8d5f3600>, signals=[15]) 2026-02-24T15:44:57.549 INFO:teuthology.lock.ops:Start node 'trial191.front.sepia.ceph.com' reimaging 2026-02-24T15:44:57.550 INFO:teuthology.lock.ops:Updating [trial191.front.sepia.ceph.com]: reset os type and version on server 2026-02-24T15:44:57.550 INFO:teuthology.lock.ops:Updating trial191.front.sepia.ceph.com on lock server 2026-02-24T15:44:57.750 DEBUG:teuthology.provision.fog.trial191:Requesting image trial_centos_9.stream (ID 3) 2026-02-24T15:44:57.796 INFO:teuthology.provision.fog.trial191:Scheduling deploy of centos 9.stream 2026-02-24T15:44:59.063 INFO:teuthology.orchestra.console.trial191:Power off 2026-02-24T15:44:59.063 DEBUG:teuthology.orchestra.console.trial191:pexpect command: ipmitool -H trial191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2026-02-24T15:44:59.102 DEBUG:teuthology.orchestra.console.trial191:power off output: Chassis Power Control: Down/Off 2026-02-24T15:45:03.106 DEBUG:teuthology.orchestra.console.trial191:pexpect command: ipmitool -H trial191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-24T15:45:03.215 DEBUG:teuthology.orchestra.console.trial191:check power output: Chassis Power is off 2026-02-24T15:45:03.216 INFO:teuthology.orchestra.console.trial191:Power off completed 2026-02-24T15:45:03.316 INFO:teuthology.orchestra.console.trial191:Power on 2026-02-24T15:45:03.317 DEBUG:teuthology.orchestra.console.trial191:pexpect command: ipmitool -H trial191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2026-02-24T15:45:03.356 DEBUG:teuthology.orchestra.console.trial191:power on output: Chassis Power Control: Up/On 2026-02-24T15:45:07.361 DEBUG:teuthology.orchestra.console.trial191:pexpect command: ipmitool -H trial191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-24T15:45:07.470 DEBUG:teuthology.orchestra.console.trial191:check power output: Chassis Power is on 2026-02-24T15:45:07.470 INFO:teuthology.orchestra.console.trial191:Power on completed 2026-02-24T15:45:07.571 INFO:teuthology.provision.fog.trial191:Waiting for deploy to finish 2026-02-24T15:46:56.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:46:56.118 ERROR:teuthology.orchestra.connection:Error authenticating with trial191.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2026-02-24T15:48:00.121 WARNING:teuthology.provision.fog.trial191:timed out 2026-02-24T15:48:06.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:48:11.523 ERROR:teuthology.orchestra.connection:Error authenticating with trial191.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2026-02-24T15:48:15.604 ERROR:teuthology.orchestra.connection:Error authenticating with trial191.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2026-02-24T15:48:22.719 INFO:teuthology.orchestra.run:Running command with timeout 600 2026-02-24T15:48:22.719 DEBUG:teuthology.orchestra.run.trial191:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done 2026-02-24T15:48:22.869 INFO:teuthology.provision.fog.trial191:Node is ready 2026-02-24T15:48:22.869 DEBUG:teuthology.orchestra.run.trial191:> hostname 2026-02-24T15:48:22.926 INFO:teuthology.orchestra.run.trial191.stdout:trial191.front.sepia.ceph.com 2026-02-24T15:48:22.926 DEBUG:teuthology.orchestra.run.trial191:> grep trial191.front.sepia.ceph.com /etc/hosts 2026-02-24T15:48:22.983 INFO:teuthology.orchestra.run.trial191.stdout:10.20.193.191 trial191.front.sepia.ceph.com trial191 2026-02-24T15:48:22.983 DEBUG:teuthology.orchestra.run.trial191:> sudo hostname trial191 2026-02-24T15:48:23.052 DEBUG:teuthology.orchestra.run.trial191:> sudo sed -i -e 's/trial191.front.sepia.ceph.com/trial191/g' /etc/hosts 2026-02-24T15:48:23.116 DEBUG:teuthology.orchestra.run.trial191:> sudo sed -i -e 's/10.20.193.191/10.20.193.191/g' /etc/hosts 2026-02-24T15:48:23.180 DEBUG:teuthology.orchestra.run.trial191:> sudo sed -i -e 's/trial191.front.sepia.ceph.com/trial191/g' /etc/hostname 2026-02-24T15:48:23.244 DEBUG:teuthology.orchestra.run.trial191:> sudo hostname trial191 2026-02-24T15:48:23.307 DEBUG:teuthology.orchestra.run.trial191:> cat /etc/os-release 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:NAME="CentOS Stream" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:VERSION="9" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:ID="centos" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:ID_LIKE="rhel fedora" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:VERSION_ID="9" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:PLATFORM_ID="platform:el9" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:ANSI_COLOR="0;31" 2026-02-24T15:48:23.363 INFO:teuthology.orchestra.run.trial191.stdout:LOGO="fedora-logo-icon" 2026-02-24T15:48:23.364 INFO:teuthology.orchestra.run.trial191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T15:48:23.364 INFO:teuthology.orchestra.run.trial191.stdout:HOME_URL="https://centos.org/" 2026-02-24T15:48:23.364 INFO:teuthology.orchestra.run.trial191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T15:48:23.364 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T15:48:23.364 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T15:48:23.364 INFO:teuthology.provision.fog.trial191:Deploy complete! 2026-02-24T15:48:23.364 DEBUG:teuthology.parallel:result is None 2026-02-24T15:48:23.583 INFO:teuthology.lock.ops:Checking trial191.front.sepia.ceph.com 2026-02-24T15:48:23.583 INFO:teuthology.lock.ops:New key found. Updating... 2026-02-24T15:48:23.652 INFO:teuthology.lock.ops:Updating [trial191.front.sepia.ceph.com]: set os type and version on server 2026-02-24T15:48:23.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-24T15:48:23.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-24T15:48:23.716 DEBUG:teuthology.orchestra.run.trial191:> true 2026-02-24T15:48:23.794 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial191.front.sepia.ceph.com' 2026-02-24T15:48:23.794 DEBUG:teuthology.orchestra.run.trial191:> uname -m 2026-02-24T15:48:23.850 INFO:teuthology.orchestra.run.trial191.stdout:x86_64 2026-02-24T15:48:23.850 DEBUG:teuthology.orchestra.run.trial191:> cat /etc/os-release 2026-02-24T15:48:23.907 INFO:teuthology.orchestra.run.trial191.stdout:NAME="CentOS Stream" 2026-02-24T15:48:23.907 INFO:teuthology.orchestra.run.trial191.stdout:VERSION="9" 2026-02-24T15:48:23.908 INFO:teuthology.orchestra.run.trial191.stdout:ID="centos" 2026-02-24T15:48:23.908 INFO:teuthology.orchestra.run.trial191.stdout:ID_LIKE="rhel fedora" 2026-02-24T15:48:23.908 INFO:teuthology.orchestra.run.trial191.stdout:VERSION_ID="9" 2026-02-24T15:48:23.908 INFO:teuthology.orchestra.run.trial191.stdout:PLATFORM_ID="platform:el9" 2026-02-24T15:48:23.908 INFO:teuthology.orchestra.run.trial191.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-24T15:48:23.909 INFO:teuthology.orchestra.run.trial191.stdout:ANSI_COLOR="0;31" 2026-02-24T15:48:23.909 INFO:teuthology.orchestra.run.trial191.stdout:LOGO="fedora-logo-icon" 2026-02-24T15:48:23.909 INFO:teuthology.orchestra.run.trial191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-24T15:48:23.909 INFO:teuthology.orchestra.run.trial191.stdout:HOME_URL="https://centos.org/" 2026-02-24T15:48:23.909 INFO:teuthology.orchestra.run.trial191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-24T15:48:23.909 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-24T15:48:23.910 INFO:teuthology.orchestra.run.trial191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-24T15:48:23.911 INFO:teuthology.lock.ops:Updating trial191.front.sepia.ceph.com on lock server 2026-02-24T15:48:24.018 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:48:24.111 INFO:teuthology.dispatcher.supervisor:Running job 68260 2026-02-24T15:48:24.111 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_liucm@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68260 --name liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial --description crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/sample_radosbench} -- /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68260/orig.config.yaml 2026-02-24T15:48:24.112 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/liucm-2026-02-24_15:28:57-crimson-rados-wip-liucm-shard-redesign-crimson-only-distro-debug-trial/68260 2026-02-24T15:48:24.112 INFO:teuthology.dispatcher.supervisor:Job PID: 1511130 2026-02-24T15:48:24.112 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2026-02-24T15:50:24.112 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:52:24.234 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:54:24.356 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:56:24.477 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T15:58:24.596 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:00:24.717 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:02:24.836 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:04:24.951 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:06:25.069 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:08:25.191 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:10:25.318 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:12:25.441 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:14:25.567 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:16:25.689 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:18:25.825 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:20:25.946 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-24T16:20:26.068 INFO:teuthology.dispatcher.supervisor:Success! 2026-02-24T16:20:26.133 INFO:teuthology.dispatcher.supervisor:Unlocking machines... 2026-02-24T16:20:26.429 INFO:teuthology.orchestra.console.trial191:Power off 2026-02-24T16:20:26.429 DEBUG:teuthology.orchestra.console.trial191:pexpect command: ipmitool -H trial191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2026-02-24T16:20:26.468 DEBUG:teuthology.orchestra.console.trial191:power off output: Chassis Power Control: Down/Off 2026-02-24T16:20:30.468 DEBUG:teuthology.orchestra.console.trial191:pexpect command: ipmitool -H trial191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-24T16:20:30.579 DEBUG:teuthology.orchestra.console.trial191:check power output: Chassis Power is on 2026-02-24T16:20:34.580 DEBUG:teuthology.orchestra.console.trial191:pexpect command: ipmitool -H trial191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-24T16:20:34.688 DEBUG:teuthology.orchestra.console.trial191:check power output: Chassis Power is off 2026-02-24T16:20:34.689 INFO:teuthology.orchestra.console.trial191:Power off completed 2026-02-24T16:20:34.887 INFO:teuthology.lock.ops:unlocked: trial191.front.sepia.ceph.com