2026-02-24T07:41:13.784 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-24T07:41:13.785 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:41:13.856 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:41:14.201 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x78afad12cc20>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x78afad0c1bc0>, signals=[15])
2026-02-24T07:41:14.201 INFO:teuthology.lock.ops:Start node 'trial072.front.sepia.ceph.com' reimaging
2026-02-24T07:41:14.202 INFO:teuthology.lock.ops:Updating [trial072.front.sepia.ceph.com]: reset os type and version on server
2026-02-24T07:41:14.202 INFO:teuthology.lock.ops:Updating trial072.front.sepia.ceph.com on lock server
2026-02-24T07:41:14.258 INFO:teuthology.lock.ops:Start node 'trial128.front.sepia.ceph.com' reimaging
2026-02-24T07:41:14.258 INFO:teuthology.lock.ops:Updating [trial128.front.sepia.ceph.com]: reset os type and version on server
2026-02-24T07:41:14.259 INFO:teuthology.lock.ops:Updating trial128.front.sepia.ceph.com on lock server
2026-02-24T07:41:14.471 DEBUG:teuthology.provision.fog.trial072:Requesting image trial_centos_9.stream (ID 3)
2026-02-24T07:41:14.501 DEBUG:teuthology.provision.fog.trial128:Requesting image trial_centos_9.stream (ID 3)
2026-02-24T07:41:14.520 INFO:teuthology.provision.fog.trial072:Scheduling deploy of centos 9.stream
2026-02-24T07:41:14.551 INFO:teuthology.provision.fog.trial128:Scheduling deploy of centos 9.stream
2026-02-24T07:41:16.176 INFO:teuthology.orchestra.console.trial072:Power off
2026-02-24T07:41:16.176 DEBUG:teuthology.orchestra.console.trial072:pexpect command: ipmitool -H trial072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T07:41:16.188 INFO:teuthology.orchestra.console.trial128:Power off
2026-02-24T07:41:16.188 DEBUG:teuthology.orchestra.console.trial128:pexpect command: ipmitool -H trial128.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T07:41:16.218 DEBUG:teuthology.orchestra.console.trial072:power off output: Chassis Power Control: Down/Off
2026-02-24T07:41:16.226 DEBUG:teuthology.orchestra.console.trial128:power off output: Chassis Power Control: Down/Off
2026-02-24T07:41:20.222 DEBUG:teuthology.orchestra.console.trial072:pexpect command: ipmitool -H trial072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:41:20.229 DEBUG:teuthology.orchestra.console.trial128:pexpect command: ipmitool -H trial128.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:41:20.331 DEBUG:teuthology.orchestra.console.trial072:check power output: Chassis Power is off
2026-02-24T07:41:20.331 INFO:teuthology.orchestra.console.trial072:Power off completed
2026-02-24T07:41:20.337 DEBUG:teuthology.orchestra.console.trial128:check power output: Chassis Power is off
2026-02-24T07:41:20.337 INFO:teuthology.orchestra.console.trial128:Power off completed
2026-02-24T07:41:20.432 INFO:teuthology.orchestra.console.trial072:Power on
2026-02-24T07:41:20.433 DEBUG:teuthology.orchestra.console.trial072:pexpect command: ipmitool -H trial072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-24T07:41:20.442 INFO:teuthology.orchestra.console.trial128:Power on
2026-02-24T07:41:20.442 DEBUG:teuthology.orchestra.console.trial128:pexpect command: ipmitool -H trial128.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-24T07:41:20.475 DEBUG:teuthology.orchestra.console.trial072:power on output: Chassis Power Control: Up/On
2026-02-24T07:41:20.487 DEBUG:teuthology.orchestra.console.trial128:power on output: Chassis Power Control: Up/On
2026-02-24T07:41:24.478 DEBUG:teuthology.orchestra.console.trial072:pexpect command: ipmitool -H trial072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:41:24.488 DEBUG:teuthology.orchestra.console.trial128:pexpect command: ipmitool -H trial128.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:41:24.590 DEBUG:teuthology.orchestra.console.trial072:check power output: Chassis Power is on
2026-02-24T07:41:24.591 INFO:teuthology.orchestra.console.trial072:Power on completed
2026-02-24T07:41:24.598 DEBUG:teuthology.orchestra.console.trial128:check power output: Chassis Power is on
2026-02-24T07:41:24.598 INFO:teuthology.orchestra.console.trial128:Power on completed
2026-02-24T07:41:24.692 INFO:teuthology.provision.fog.trial072:Waiting for deploy to finish
2026-02-24T07:41:24.699 INFO:teuthology.provision.fog.trial128:Waiting for deploy to finish
2026-02-24T07:43:12.766 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:43:28.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial128.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:43:31.200 WARNING:teuthology.provision.fog.trial128:[Errno None] Unable to connect to port 22 on 10.20.193.128
2026-02-24T07:43:37.206 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial128.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:43:40.287 WARNING:teuthology.provision.fog.trial128:[Errno None] Unable to connect to port 22 on 10.20.193.128
2026-02-24T07:43:46.294 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial128.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:43:49.377 WARNING:teuthology.provision.fog.trial128:[Errno None] Unable to connect to port 22 on 10.20.193.128
2026-02-24T07:43:55.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial128.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:43:58.464 WARNING:teuthology.provision.fog.trial128:[Errno None] Unable to connect to port 22 on 10.20.193.128
2026-02-24T07:44:04.470 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial128.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:44:12.775 WARNING:teuthology.provision.fog.trial072:timed out
2026-02-24T07:44:18.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:44:26.106 ERROR:teuthology.orchestra.connection:Error authenticating with trial072.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-24T07:44:30.195 ERROR:teuthology.orchestra.connection:Error authenticating with trial072.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/connection.py", line 111, in connect
    ssh.connect(**connect_args)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/client.py", line 794, in _auth
    self._transport.auth_publickey(username, key)
  File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/paramiko/transport.py", line 1664, in auth_publickey
    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
2026-02-24T07:44:37.288 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-24T07:44:37.289 DEBUG:teuthology.orchestra.run.trial072:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-24T07:44:37.432 INFO:teuthology.provision.fog.trial072:Node is ready
2026-02-24T07:44:37.432 DEBUG:teuthology.orchestra.run.trial072:> hostname
2026-02-24T07:44:37.488 INFO:teuthology.orchestra.run.trial072.stdout:trial072.front.sepia.ceph.com
2026-02-24T07:44:37.489 DEBUG:teuthology.orchestra.run.trial072:> grep trial072.front.sepia.ceph.com /etc/hosts
2026-02-24T07:44:37.544 INFO:teuthology.orchestra.run.trial072.stdout:10.20.193.72 trial072.front.sepia.ceph.com trial072
2026-02-24T07:44:37.545 DEBUG:teuthology.orchestra.run.trial072:> sudo hostname trial072
2026-02-24T07:44:37.615 DEBUG:teuthology.orchestra.run.trial072:> sudo sed -i -e 's/trial072.front.sepia.ceph.com/trial072/g' /etc/hosts
2026-02-24T07:44:37.681 DEBUG:teuthology.orchestra.run.trial072:> sudo sed -i -e 's/10.20.193.72/10.20.193.72/g' /etc/hosts
2026-02-24T07:44:37.747 DEBUG:teuthology.orchestra.run.trial072:> sudo sed -i -e 's/trial072.front.sepia.ceph.com/trial072/g' /etc/hostname
2026-02-24T07:44:37.812 DEBUG:teuthology.orchestra.run.trial072:> sudo hostname trial072
2026-02-24T07:44:37.877 DEBUG:teuthology.orchestra.run.trial072:> cat /etc/os-release
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:NAME="CentOS Stream"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:VERSION="9"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:ID="centos"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:VERSION_ID="9"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:ANSI_COLOR="0;31"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:44:37.933 INFO:teuthology.orchestra.run.trial072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:44:37.934 INFO:teuthology.provision.fog.trial072:Deploy complete!
2026-02-24T07:44:37.934 DEBUG:teuthology.parallel:result is None
2026-02-24T07:44:40.414 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-24T07:44:40.414 DEBUG:teuthology.orchestra.run.trial128:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-24T07:44:40.569 INFO:teuthology.provision.fog.trial128:Node is ready
2026-02-24T07:44:40.569 DEBUG:teuthology.orchestra.run.trial128:> hostname
2026-02-24T07:44:40.624 INFO:teuthology.orchestra.run.trial128.stdout:trial128.front.sepia.ceph.com
2026-02-24T07:44:40.624 DEBUG:teuthology.orchestra.run.trial128:> grep trial128.front.sepia.ceph.com /etc/hosts
2026-02-24T07:44:40.681 INFO:teuthology.orchestra.run.trial128.stdout:10.20.193.128 trial128.front.sepia.ceph.com trial128
2026-02-24T07:44:40.681 DEBUG:teuthology.orchestra.run.trial128:> sudo hostname trial128
2026-02-24T07:44:40.750 DEBUG:teuthology.orchestra.run.trial128:> sudo sed -i -e 's/trial128.front.sepia.ceph.com/trial128/g' /etc/hosts
2026-02-24T07:44:40.814 DEBUG:teuthology.orchestra.run.trial128:> sudo sed -i -e 's/10.20.193.128/10.20.193.128/g' /etc/hosts
2026-02-24T07:44:40.880 DEBUG:teuthology.orchestra.run.trial128:> sudo sed -i -e 's/trial128.front.sepia.ceph.com/trial128/g' /etc/hostname
2026-02-24T07:44:40.944 DEBUG:teuthology.orchestra.run.trial128:> sudo hostname trial128
2026-02-24T07:44:41.010 DEBUG:teuthology.orchestra.run.trial128:> cat /etc/os-release
2026-02-24T07:44:41.065 INFO:teuthology.orchestra.run.trial128.stdout:NAME="CentOS Stream"
2026-02-24T07:44:41.065 INFO:teuthology.orchestra.run.trial128.stdout:VERSION="9"
2026-02-24T07:44:41.065 INFO:teuthology.orchestra.run.trial128.stdout:ID="centos"
2026-02-24T07:44:41.065 INFO:teuthology.orchestra.run.trial128.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:VERSION_ID="9"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:ANSI_COLOR="0;31"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:44:41.066 INFO:teuthology.orchestra.run.trial128.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:44:41.066 INFO:teuthology.provision.fog.trial128:Deploy complete!
2026-02-24T07:44:41.066 DEBUG:teuthology.parallel:result is None
2026-02-24T07:44:41.422 INFO:teuthology.lock.ops:Checking trial072.front.sepia.ceph.com
2026-02-24T07:44:41.423 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-24T07:44:41.508 INFO:teuthology.lock.ops:Checking trial128.front.sepia.ceph.com
2026-02-24T07:44:41.508 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-24T07:44:41.569 INFO:teuthology.lock.ops:Updating [trial072.front.sepia.ceph.com]: set os type and version on server
2026-02-24T07:44:41.569 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial072.front.sepia.ceph.com'
2026-02-24T07:44:41.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:44:41.632 DEBUG:teuthology.orchestra.run.trial072:> true
2026-02-24T07:44:41.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial072.front.sepia.ceph.com'
2026-02-24T07:44:41.708 DEBUG:teuthology.orchestra.run.trial072:> uname -m
2026-02-24T07:44:41.763 INFO:teuthology.orchestra.run.trial072.stdout:x86_64
2026-02-24T07:44:41.763 DEBUG:teuthology.orchestra.run.trial072:> cat /etc/os-release
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:NAME="CentOS Stream"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:VERSION="9"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:ID="centos"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:VERSION_ID="9"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:ANSI_COLOR="0;31"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:44:41.818 INFO:teuthology.orchestra.run.trial072.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:44:41.819 INFO:teuthology.orchestra.run.trial072.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:44:41.819 INFO:teuthology.orchestra.run.trial072.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:44:41.819 INFO:teuthology.orchestra.run.trial072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:44:41.819 INFO:teuthology.orchestra.run.trial072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:44:41.819 INFO:teuthology.lock.ops:Updating trial072.front.sepia.ceph.com on lock server
2026-02-24T07:44:41.882 INFO:teuthology.lock.ops:Updating [trial128.front.sepia.ceph.com]: set os type and version on server
2026-02-24T07:44:41.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial128.front.sepia.ceph.com'
2026-02-24T07:44:41.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial128.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:44:41.944 DEBUG:teuthology.orchestra.run.trial128:> true
2026-02-24T07:44:42.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial128.front.sepia.ceph.com'
2026-02-24T07:44:42.017 DEBUG:teuthology.orchestra.run.trial128:> uname -m
2026-02-24T07:44:42.073 INFO:teuthology.orchestra.run.trial128.stdout:x86_64
2026-02-24T07:44:42.073 DEBUG:teuthology.orchestra.run.trial128:> cat /etc/os-release
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:NAME="CentOS Stream"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:VERSION="9"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:ID="centos"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:VERSION_ID="9"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:ANSI_COLOR="0;31"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:44:42.129 INFO:teuthology.orchestra.run.trial128.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:44:42.130 INFO:teuthology.lock.ops:Updating trial128.front.sepia.ceph.com on lock server
2026-02-24T07:44:42.195 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:44:42.285 INFO:teuthology.dispatcher.supervisor:Running job 67594
2026-02-24T07:44:42.285 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_shraddhaag@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594 --name shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial --description crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/cephadm objectstore/crimson_bluestore thrashers/{default} thrashosds-health workloads/snaps-few-objects-balanced} -- /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594/orig.config.yaml
2026-02-24T07:44:42.286 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594
2026-02-24T07:44:42.286 INFO:teuthology.dispatcher.supervisor:Job PID: 2944557
2026-02-24T07:44:42.286 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-24T07:46:42.386 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:48:42.553 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:50:42.731 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:52:42.914 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:54:43.048 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:56:43.126 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:58:43.286 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:00:43.424 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:02:43.623 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:04:43.815 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:06:44.037 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:08:44.161 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:10:44.368 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:12:44.504 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:14:44.670 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:16:44.862 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:18:49.189 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:20:49.276 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:22:49.478 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:24:49.654 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:26:49.780 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:28:49.988 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:30:50.176 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:32:50.386 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:34:50.542 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:36:50.779 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:38:50.955 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:40:51.161 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:42:51.296 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:44:51.430 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:46:51.568 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:48:51.764 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:50:51.990 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:52:52.203 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:54:52.406 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:56:52.590 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:58:52.708 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:00:52.909 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:02:53.141 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:04:53.351 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:06:53.645 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:08:53.827 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:10:54.019 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:12:54.207 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:14:54.389 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:16:54.581 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:18:54.771 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:20:54.980 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:22:55.134 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:24:55.304 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:26:55.508 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:28:55.671 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:30:55.837 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:32:56.010 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:34:56.229 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:36:56.438 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:38:56.564 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:40:56.781 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:42:56.977 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:44:57.134 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:46:57.314 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:48:57.520 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:50:57.737 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:52:57.865 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:54:58.063 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:56:58.265 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:58:58.489 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:00:58.688 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:02:58.872 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:04:59.061 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:06:59.207 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:08:59.417 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:10:59.627 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:12:59.813 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:15:00.020 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:17:00.268 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:19:00.414 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:21:00.543 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:23:00.747 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:25:00.957 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:27:01.120 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:29:01.295 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:31:01.472 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:33:01.596 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:35:01.761 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:37:01.902 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:39:02.078 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:41:04.330 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:43:04.548 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:45:04.756 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:47:04.958 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:49:05.119 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:51:05.343 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:53:05.472 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:55:05.623 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:57:05.828 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:59:06.027 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:01:06.231 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:03:06.420 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:05:06.584 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:07:06.781 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:09:06.860 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:11:07.024 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:13:07.145 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:15:07.360 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:17:07.569 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:19:07.738 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:21:07.908 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:23:08.068 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:25:08.259 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:27:08.462 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:29:08.664 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:31:08.817 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:33:08.930 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:35:09.113 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:37:09.313 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:39:09.502 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:41:09.636 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:43:09.824 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:45:10.022 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:47:10.203 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:49:10.380 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:51:10.514 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:53:10.673 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:55:10.852 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:57:11.037 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:59:11.165 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:01:11.336 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:03:11.497 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:05:11.708 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:07:11.880 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:09:12.034 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:11:12.227 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:13:12.407 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:15:12.587 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:17:12.800 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:19:12.893 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:21:13.098 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:23:13.298 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:25:13.507 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:27:13.713 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:29:13.923 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:31:14.156 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:33:14.354 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:35:14.550 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:37:14.656 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:39:14.840 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:41:14.943 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:43:15.149 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:45:15.327 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:47:15.533 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:49:15.719 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:51:15.848 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:53:16.059 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:55:16.221 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:57:16.446 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:59:16.660 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:01:16.872 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:03:17.087 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:05:17.336 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:07:17.448 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:09:17.582 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:11:17.755 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:13:17.917 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:15:18.092 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:17:18.293 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:19:18.511 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:21:18.724 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:23:18.797 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:25:18.997 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:27:19.180 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:29:19.335 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:31:19.444 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:33:19.615 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:35:19.782 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:37:19.984 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:39:20.187 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:41:20.383 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:43:20.491 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:45:20.651 WARNING:teuthology.dispatcher.supervisor:Job shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594 ran longer than 21600s. Killing...
2026-02-24T13:45:20.732 INFO:teuthology.kill:Killing Pids: {2944557}
2026-02-24T13:45:20.733 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:45:20.841 INFO:teuthology.task.internal:roles: ubuntu@trial072.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a']
2026-02-24T13:45:20.842 INFO:teuthology.task.internal:roles: ubuntu@trial128.front.sepia.ceph.com - ['osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']
2026-02-24T13:45:20.842 INFO:teuthology.misc:Compressing logs...
2026-02-24T13:45:20.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial072.front.sepia.ceph.com'
2026-02-24T13:45:20.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T13:45:20.912 DEBUG:teuthology.orchestra.run.trial072:> true
2026-02-24T13:45:20.990 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial072.front.sepia.ceph.com'
2026-02-24T13:45:20.990 DEBUG:teuthology.orchestra.run.trial072:> sudo find /var/lib/ceph/crash -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:45:21.032 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial128.front.sepia.ceph.com'
2026-02-24T13:45:21.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial128.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T13:45:21.135 DEBUG:teuthology.orchestra.run.trial128:> true
2026-02-24T13:45:21.214 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial128.front.sepia.ceph.com'
2026-02-24T13:45:21.214 DEBUG:teuthology.orchestra.run.trial128:> sudo find /var/lib/ceph/crash -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:45:21.284 DEBUG:teuthology.misc:Transferring archived files from trial072:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594/remote/trial072/crash
2026-02-24T13:45:21.284 DEBUG:teuthology.orchestra.run.trial072:> sudo tar c -f - -C /var/lib/ceph/crash -- .
2026-02-24T13:45:21.312 DEBUG:teuthology.misc:Transferring archived files from trial128:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594/remote/trial128/crash
2026-02-24T13:45:21.313 DEBUG:teuthology.orchestra.run.trial128:> sudo tar c -f - -C /var/lib/ceph/crash -- .
2026-02-24T13:45:21.347 INFO:teuthology.misc:Compressing logs...
2026-02-24T13:45:21.347 DEBUG:teuthology.orchestra.run.trial072:> sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:45:21.356 DEBUG:teuthology.orchestra.run.trial128:> sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:45:21.376 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2026-02-24T13:45:21.378 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2026-02-24T13:45:21.378 INFO:teuthology.orchestra.run.trial072.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2026-02-24T13:45:21.378 INFO:teuthology.orchestra.run.trial072.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2026-02-24T13:45:21.411 INFO:teuthology.orchestra.run.trial128.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2026-02-24T13:45:21.412 INFO:teuthology.orchestra.run.trial128.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2026-02-24T13:45:21.412 INFO:teuthology.orchestra.run.trial128.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2026-02-24T13:45:21.413 INFO:teuthology.orchestra.run.trial128.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2026-02-24T13:45:21.414 DEBUG:teuthology.misc:Transferring archived files from trial072:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594/remote/trial072/
2026-02-24T13:45:21.414 DEBUG:teuthology.orchestra.run.trial072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2026-02-24T13:45:21.447 DEBUG:teuthology.misc:Transferring archived files from trial128:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594/remote/trial128/
2026-02-24T13:45:21.447 DEBUG:teuthology.orchestra.run.trial128:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2026-02-24T13:45:21.480 INFO:teuthology.misc:Compressing logs...
2026-02-24T13:45:21.481 DEBUG:teuthology.orchestra.run.trial072:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:45:21.492 DEBUG:teuthology.orchestra.run.trial128:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:45:21.512 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log
2026-02-24T13:45:21.514 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mon.a.log
2026-02-24T13:45:21.514 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/cephadm.log:	gzip -5 --verbose -- /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.cephadm.log
2026-02-24T13:45:21.514 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.audit.log
2026-02-24T13:45:21.515 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mon.a.log:	/var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.cephadm.log:	gzip -5 --verbose -- /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-volume.log
2026-02-24T13:45:21.515 INFO:teuthology.orchestra.run.trial072.stderr: 81.9% -- replaced with /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.cephadm.log.gz
2026-02-24T13:45:21.515 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.audit.log:	gzip -5 --verbose -- /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mgr.x.log
2026-02-24T13:45:21.515 INFO:teuthology.orchestra.run.trial072.stderr:gzip -5 --verbose -- /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.log
2026-02-24T13:45:21.522 INFO:teuthology.orchestra.run.trial072.stderr:/var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-volume.log:	/var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mgr.x.log:	/var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.log:	 90.5% -- replaced with /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.audit.log.gz
2026-02-24T13:45:21.528 INFO:teuthology.orchestra.run.trial072.stderr: 91.9% -- replaced with /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph.log.gz
2026-02-24T13:45:21.528 INFO:teuthology.orchestra.run.trial072.stderr: 92.5% -- replaced with /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-volume.log.gz
2026-02-24T13:45:21.546 INFO:teuthology.orchestra.run.trial128.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log
2026-02-24T13:45:21.547 INFO:teuthology.orchestra.run.trial128.stderr:gzip -5 --verbose -- /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-volume.log
2026-02-24T13:45:21.549 INFO:teuthology.orchestra.run.trial128.stderr:/var/log/ceph/cephadm.log:	/var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-volume.log:	 86.7% -- replaced with /var/log/ceph/cephadm.log.gz
2026-02-24T13:45:21.550 INFO:teuthology.orchestra.run.trial072.stderr: 97.0% -- replaced with /var/log/ceph/cephadm.log.gz
2026-02-24T13:45:21.561 INFO:teuthology.orchestra.run.trial128.stderr: 92.4% -- replaced with /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-volume.log.gz
2026-02-24T13:45:22.135 INFO:teuthology.orchestra.run.trial072.stderr:gzip: /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mgr.x.log: file size changed while zipping
2026-02-24T13:45:22.135 INFO:teuthology.orchestra.run.trial072.stderr: 92.4% -- replaced with /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mgr.x.log.gz
2026-02-24T13:45:25.457 INFO:teuthology.orchestra.run.trial072.stderr:gzip: /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mon.a.log: file size changed while zipping
2026-02-24T13:45:25.458 INFO:teuthology.orchestra.run.trial072.stderr: 88.8% -- replaced with /var/log/ceph/e410d078-1159-11f1-b7b2-d404e6e7d460/ceph-mon.a.log.gz
2026-02-24T13:45:25.460 DEBUG:teuthology.misc:Transferring archived files from trial072:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594/remote/trial072/log
2026-02-24T13:45:25.461 DEBUG:teuthology.orchestra.run.trial072:> sudo tar c -f - -C /var/log/ceph -- .
2026-02-24T13:45:26.052 DEBUG:teuthology.misc:Transferring archived files from trial128:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67594/remote/trial128/log
2026-02-24T13:45:26.055 DEBUG:teuthology.orchestra.run.trial128:> sudo tar c -f - -C /var/log/ceph -- .
2026-02-24T13:45:26.170 INFO:teuthology.kill:No teuthology processes running
2026-02-24T13:45:26.171 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:45:26.834 INFO:teuthology.orchestra.console.trial128:Power off
2026-02-24T13:45:26.834 DEBUG:teuthology.orchestra.console.trial128:pexpect command: ipmitool -H trial128.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T13:45:26.845 INFO:teuthology.orchestra.console.trial072:Power off
2026-02-24T13:45:26.845 DEBUG:teuthology.orchestra.console.trial072:pexpect command: ipmitool -H trial072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T13:45:26.875 DEBUG:teuthology.orchestra.console.trial128:power off output: Chassis Power Control: Down/Off
2026-02-24T13:45:26.884 DEBUG:teuthology.orchestra.console.trial072:power off output: Chassis Power Control: Down/Off
2026-02-24T13:45:30.879 DEBUG:teuthology.orchestra.console.trial128:pexpect command: ipmitool -H trial128.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:45:30.886 DEBUG:teuthology.orchestra.console.trial072:pexpect command: ipmitool -H trial072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:45:30.989 DEBUG:teuthology.orchestra.console.trial128:check power output: Chassis Power is on
2026-02-24T13:45:30.995 DEBUG:teuthology.orchestra.console.trial072:check power output: Chassis Power is on
2026-02-24T13:45:34.993 DEBUG:teuthology.orchestra.console.trial128:pexpect command: ipmitool -H trial128.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:45:35.004 DEBUG:teuthology.orchestra.console.trial072:pexpect command: ipmitool -H trial072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:45:35.106 DEBUG:teuthology.orchestra.console.trial128:check power output: Chassis Power is off
2026-02-24T13:45:35.106 INFO:teuthology.orchestra.console.trial128:Power off completed
2026-02-24T13:45:35.114 DEBUG:teuthology.orchestra.console.trial072:check power output: Chassis Power is off
2026-02-24T13:45:35.114 INFO:teuthology.orchestra.console.trial072:Power off completed
2026-02-24T13:45:35.359 INFO:teuthology.lock.ops:unlocked: trial128.front.sepia.ceph.com
2026-02-24T13:45:35.372 INFO:teuthology.lock.ops:unlocked: trial072.front.sepia.ceph.com
2026-02-24T13:45:35.373 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:47:35.542 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:47:35.613 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2026-02-24T13:47:35.737 WARNING:teuthology.dispatcher.supervisor:Was going to unlock trial072 but it was locked by another job: /home/teuthworker/mnt/teuthology/cjames-2026-02-24_10:01:51-rados-main-distro-default-trial/67846
2026-02-24T13:47:35.737 WARNING:teuthology.dispatcher.supervisor:Was going to unlock trial128 but it was locked by another job: /home/teuthworker/mnt/teuthology/cjames-2026-02-24_10:01:51-rados-main-distro-default-trial/67846