2026-02-25T09:02:22.581 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T09:02:22.582 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:02:22.668 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:02:22.802 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7e134b6be660>, signals=[15]) 2026-02-25T09:02:22.802 INFO:teuthology.lock.ops:Start node 'trial067.front.sepia.ceph.com' reimaging 2026-02-25T09:02:22.802 INFO:teuthology.lock.ops:Updating [trial067.front.sepia.ceph.com]: reset os type and version on server 2026-02-25T09:02:22.802 INFO:teuthology.lock.ops:Updating trial067.front.sepia.ceph.com on lock server 2026-02-25T09:02:23.008 DEBUG:teuthology.provision.fog.trial067:Requesting image trial_centos_9.stream (ID 3) 2026-02-25T09:02:23.059 INFO:teuthology.provision.fog.trial067:Scheduling deploy of centos 9.stream 2026-02-25T09:02:24.180 INFO:teuthology.orchestra.console.trial067:Power off 2026-02-25T09:02:24.180 DEBUG:teuthology.orchestra.console.trial067:pexpect command: ipmitool -H trial067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2026-02-25T09:02:24.220 DEBUG:teuthology.orchestra.console.trial067:power off output: Chassis Power Control: Down/Off 2026-02-25T09:02:28.224 DEBUG:teuthology.orchestra.console.trial067:pexpect command: ipmitool -H trial067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-25T09:02:28.333 DEBUG:teuthology.orchestra.console.trial067:check power output: Chassis Power is off 2026-02-25T09:02:28.333 INFO:teuthology.orchestra.console.trial067:Power off completed 2026-02-25T09:02:28.434 INFO:teuthology.orchestra.console.trial067:Power on 2026-02-25T09:02:28.434 DEBUG:teuthology.orchestra.console.trial067:pexpect command: ipmitool -H trial067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2026-02-25T09:02:28.476 DEBUG:teuthology.orchestra.console.trial067:power on output: Chassis Power Control: Up/On 2026-02-25T09:02:32.481 DEBUG:teuthology.orchestra.console.trial067:pexpect command: ipmitool -H trial067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-25T09:02:32.590 DEBUG:teuthology.orchestra.console.trial067:check power output: Chassis Power is on 2026-02-25T09:02:32.590 INFO:teuthology.orchestra.console.trial067:Power on completed 2026-02-25T09:02:32.691 INFO:teuthology.provision.fog.trial067:Waiting for deploy to finish 2026-02-25T09:04:20.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T09:04:20.467 ERROR:teuthology.orchestra.connection:Error authenticating with trial067.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2026-02-25T09:05:24.470 WARNING:teuthology.provision.fog.trial067:timed out 2026-02-25T09:05:30.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T09:05:33.709 ERROR:teuthology.orchestra.connection:Error authenticating with trial067.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2026-02-25T09:05:37.843 ERROR:teuthology.orchestra.connection:Error authenticating with trial067.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2026-02-25T09:05:44.959 INFO:teuthology.orchestra.run:Running command with timeout 600 2026-02-25T09:05:44.959 DEBUG:teuthology.orchestra.run.trial067:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done 2026-02-25T09:05:45.124 INFO:teuthology.provision.fog.trial067:Node is ready 2026-02-25T09:05:45.124 DEBUG:teuthology.orchestra.run.trial067:> hostname 2026-02-25T09:05:45.180 INFO:teuthology.orchestra.run.trial067.stdout:trial067.front.sepia.ceph.com 2026-02-25T09:05:45.181 DEBUG:teuthology.orchestra.run.trial067:> grep trial067.front.sepia.ceph.com /etc/hosts 2026-02-25T09:05:45.235 INFO:teuthology.orchestra.run.trial067.stdout:10.20.193.67 trial067.front.sepia.ceph.com trial067 2026-02-25T09:05:45.236 DEBUG:teuthology.orchestra.run.trial067:> sudo hostname trial067 2026-02-25T09:05:45.314 DEBUG:teuthology.orchestra.run.trial067:> sudo sed -i -e 's/trial067.front.sepia.ceph.com/trial067/g' /etc/hosts 2026-02-25T09:05:45.379 DEBUG:teuthology.orchestra.run.trial067:> sudo sed -i -e 's/10.20.193.67/10.20.193.67/g' /etc/hosts 2026-02-25T09:05:45.446 DEBUG:teuthology.orchestra.run.trial067:> sudo sed -i -e 's/trial067.front.sepia.ceph.com/trial067/g' /etc/hostname 2026-02-25T09:05:45.512 DEBUG:teuthology.orchestra.run.trial067:> sudo hostname trial067 2026-02-25T09:05:45.577 DEBUG:teuthology.orchestra.run.trial067:> cat /etc/os-release 2026-02-25T09:05:45.633 INFO:teuthology.orchestra.run.trial067.stdout:NAME="CentOS Stream" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:VERSION="9" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:ID="centos" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:ID_LIKE="rhel fedora" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:VERSION_ID="9" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:PLATFORM_ID="platform:el9" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:ANSI_COLOR="0;31" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:LOGO="fedora-logo-icon" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:HOME_URL="https://centos.org/" 2026-02-25T09:05:45.634 INFO:teuthology.orchestra.run.trial067.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T09:05:45.635 INFO:teuthology.orchestra.run.trial067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T09:05:45.635 INFO:teuthology.orchestra.run.trial067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T09:05:45.636 INFO:teuthology.provision.fog.trial067:Deploy complete! 2026-02-25T09:05:45.636 DEBUG:teuthology.parallel:result is None 2026-02-25T09:05:45.928 INFO:teuthology.lock.ops:Checking trial067.front.sepia.ceph.com 2026-02-25T09:05:45.929 INFO:teuthology.lock.ops:New key found. Updating... 2026-02-25T09:05:46.022 INFO:teuthology.lock.ops:Updating [trial067.front.sepia.ceph.com]: set os type and version on server 2026-02-25T09:05:46.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial067.front.sepia.ceph.com' 2026-02-25T09:05:46.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T09:05:46.086 DEBUG:teuthology.orchestra.run.trial067:> true 2026-02-25T09:05:46.156 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial067.front.sepia.ceph.com' 2026-02-25T09:05:46.156 DEBUG:teuthology.orchestra.run.trial067:> uname -m 2026-02-25T09:05:46.211 INFO:teuthology.orchestra.run.trial067.stdout:x86_64 2026-02-25T09:05:46.212 DEBUG:teuthology.orchestra.run.trial067:> cat /etc/os-release 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:NAME="CentOS Stream" 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:VERSION="9" 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:ID="centos" 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:ID_LIKE="rhel fedora" 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:VERSION_ID="9" 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:PLATFORM_ID="platform:el9" 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T09:05:46.269 INFO:teuthology.orchestra.run.trial067.stdout:ANSI_COLOR="0;31" 2026-02-25T09:05:46.270 INFO:teuthology.orchestra.run.trial067.stdout:LOGO="fedora-logo-icon" 2026-02-25T09:05:46.270 INFO:teuthology.orchestra.run.trial067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T09:05:46.270 INFO:teuthology.orchestra.run.trial067.stdout:HOME_URL="https://centos.org/" 2026-02-25T09:05:46.270 INFO:teuthology.orchestra.run.trial067.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T09:05:46.270 INFO:teuthology.orchestra.run.trial067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T09:05:46.270 INFO:teuthology.orchestra.run.trial067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T09:05:46.271 INFO:teuthology.lock.ops:Updating trial067.front.sepia.ceph.com on lock server 2026-02-25T09:05:46.373 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:05:46.464 INFO:teuthology.dispatcher.supervisor:Running job 69213 2026-02-25T09:05:46.464 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_matan@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:58:25-crimson-rados-main-distro-debug-trial/69213 --name matan-2026-02-25_07:58:25-crimson-rados-main-distro-debug-trial --description crimson-rados/perf/{0-crimson_install clusters/{crimson-fixed-1} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm_2q} perf_ignorelist settings/optimized workloads/sample_radosbench} -- /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:58:25-crimson-rados-main-distro-debug-trial/69213/orig.config.yaml 2026-02-25T09:05:46.464 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:58:25-crimson-rados-main-distro-debug-trial/69213 2026-02-25T09:05:46.464 INFO:teuthology.dispatcher.supervisor:Job PID: 3794737 2026-02-25T09:05:46.464 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2026-02-25T09:07:46.465 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:09:46.587 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:11:46.676 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:13:46.799 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:15:46.921 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:17:47.042 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:19:47.162 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:21:47.286 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:23:47.407 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:25:47.527 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:27:47.650 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T09:27:47.781 INFO:teuthology.dispatcher.supervisor:Success! 2026-02-25T09:27:47.847 INFO:teuthology.dispatcher.supervisor:Unlocking machines... 2026-02-25T09:27:48.145 INFO:teuthology.orchestra.console.trial067:Power off 2026-02-25T09:27:48.145 DEBUG:teuthology.orchestra.console.trial067:pexpect command: ipmitool -H trial067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2026-02-25T09:27:48.183 DEBUG:teuthology.orchestra.console.trial067:power off output: Chassis Power Control: Down/Off 2026-02-25T09:27:52.183 DEBUG:teuthology.orchestra.console.trial067:pexpect command: ipmitool -H trial067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-25T09:27:52.295 DEBUG:teuthology.orchestra.console.trial067:check power output: Chassis Power is on 2026-02-25T09:27:56.296 DEBUG:teuthology.orchestra.console.trial067:pexpect command: ipmitool -H trial067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2026-02-25T09:27:56.409 DEBUG:teuthology.orchestra.console.trial067:check power output: Chassis Power is off 2026-02-25T09:27:56.409 INFO:teuthology.orchestra.console.trial067:Power off completed 2026-02-25T09:27:56.625 INFO:teuthology.lock.ops:unlocked: trial067.front.sepia.ceph.com