2026-02-12T11:04:14.601 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71
2026-02-12T11:04:14.601 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:04:14.674 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:04:14.865 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7529e8babb30>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7529e8393740>, signals=[15])
2026-02-12T11:04:14.865 INFO:teuthology.lock.ops:Start node 'trial043.front.sepia.ceph.com' reimaging
2026-02-12T11:04:14.865 INFO:teuthology.lock.ops:Updating [trial043.front.sepia.ceph.com]: reset os type and version on server
2026-02-12T11:04:14.865 INFO:teuthology.lock.ops:Updating trial043.front.sepia.ceph.com on lock server
2026-02-12T11:04:14.922 INFO:teuthology.lock.ops:Start node 'trial123.front.sepia.ceph.com' reimaging
2026-02-12T11:04:14.922 INFO:teuthology.lock.ops:Updating [trial123.front.sepia.ceph.com]: reset os type and version on server
2026-02-12T11:04:14.922 INFO:teuthology.lock.ops:Updating trial123.front.sepia.ceph.com on lock server
2026-02-12T11:04:15.142 DEBUG:teuthology.provision.fog.trial123:Requesting image trial_centos_9.stream (ID 3)
2026-02-12T11:04:15.150 DEBUG:teuthology.provision.fog.trial043:Requesting image trial_centos_9.stream (ID 3)
2026-02-12T11:04:15.193 INFO:teuthology.provision.fog.trial123:Scheduling deploy of centos 9.stream
2026-02-12T11:04:15.195 INFO:teuthology.provision.fog.trial043:Scheduling deploy of centos 9.stream
2026-02-12T11:04:16.077 INFO:teuthology.orchestra.console.trial123:Power off
2026-02-12T11:04:16.077 DEBUG:teuthology.orchestra.console.trial123:pexpect command: ipmitool -H trial123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T11:04:16.145 DEBUG:teuthology.orchestra.console.trial123:power off output: Chassis Power Control: Down/Off
2026-02-12T11:04:16.158 INFO:teuthology.orchestra.console.trial043:Power off
2026-02-12T11:04:16.158 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T11:04:16.197 DEBUG:teuthology.orchestra.console.trial043:power off output: Chassis Power Control: Down/Off
2026-02-12T11:04:20.150 DEBUG:teuthology.orchestra.console.trial123:pexpect command: ipmitool -H trial123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T11:04:20.198 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T11:04:20.259 DEBUG:teuthology.orchestra.console.trial123:check power output: Chassis Power is off
2026-02-12T11:04:20.259 INFO:teuthology.orchestra.console.trial123:Power off completed
2026-02-12T11:04:20.305 DEBUG:teuthology.orchestra.console.trial043:check power output: Chassis Power is off
2026-02-12T11:04:20.306 INFO:teuthology.orchestra.console.trial043:Power off completed
2026-02-12T11:04:20.360 INFO:teuthology.orchestra.console.trial123:Power on
2026-02-12T11:04:20.360 DEBUG:teuthology.orchestra.console.trial123:pexpect command: ipmitool -H trial123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-12T11:04:20.405 DEBUG:teuthology.orchestra.console.trial123:power on output: Chassis Power Control: Up/On
2026-02-12T11:04:20.407 INFO:teuthology.orchestra.console.trial043:Power on
2026-02-12T11:04:20.407 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-12T11:04:20.445 DEBUG:teuthology.orchestra.console.trial043:power on output: Chassis Power Control: Up/On
2026-02-12T11:04:24.410 DEBUG:teuthology.orchestra.console.trial123:pexpect command: ipmitool -H trial123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T11:04:24.446 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T11:04:24.522 DEBUG:teuthology.orchestra.console.trial123:check power output: Chassis Power is on
2026-02-12T11:04:24.522 INFO:teuthology.orchestra.console.trial123:Power on completed
2026-02-12T11:04:24.557 DEBUG:teuthology.orchestra.console.trial043:check power output: Chassis Power is off
2026-02-12T11:04:24.624 INFO:teuthology.provision.fog.trial123:Waiting for deploy to finish
2026-02-12T11:04:28.562 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T11:04:28.672 DEBUG:teuthology.orchestra.console.trial043:check power output: Chassis Power is on
2026-02-12T11:04:28.672 INFO:teuthology.orchestra.console.trial043:Power on completed
2026-02-12T11:04:28.773 INFO:teuthology.provision.fog.trial043:Waiting for deploy to finish
2026-02-12T11:06:12.020 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:15.133 WARNING:teuthology.provision.fog.trial123:[Errno None] Unable to connect to port 22 on 10.20.193.123
2026-02-12T11:06:15.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:18.973 WARNING:teuthology.provision.fog.trial043:[Errno None] Unable to connect to port 22 on 10.20.193.43
2026-02-12T11:06:21.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:24.221 WARNING:teuthology.provision.fog.trial123:[Errno None] Unable to connect to port 22 on 10.20.193.123
2026-02-12T11:06:24.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:28.061 WARNING:teuthology.provision.fog.trial043:[Errno None] Unable to connect to port 22 on 10.20.193.43
2026-02-12T11:06:30.223 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:33.309 WARNING:teuthology.provision.fog.trial123:[Errno None] Unable to connect to port 22 on 10.20.193.123
2026-02-12T11:06:34.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:37.149 WARNING:teuthology.provision.fog.trial043:[Errno None] Unable to connect to port 22 on 10.20.193.43
2026-02-12T11:06:39.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:42.397 WARNING:teuthology.provision.fog.trial123:[Errno None] Unable to connect to port 22 on 10.20.193.123
2026-02-12T11:06:43.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:46.237 WARNING:teuthology.provision.fog.trial043:[Errno None] Unable to connect to port 22 on 10.20.193.43
2026-02-12T11:06:48.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:06:52.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:07:24.311 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-12T11:07:24.311 DEBUG:teuthology.orchestra.run.trial123:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-12T11:07:24.547 INFO:teuthology.provision.fog.trial123:Node is ready
2026-02-12T11:07:24.547 DEBUG:teuthology.orchestra.run.trial123:> hostname
2026-02-12T11:07:24.603 INFO:teuthology.orchestra.run.trial123.stdout:trial123.front.sepia.ceph.com
2026-02-12T11:07:24.604 DEBUG:teuthology.orchestra.run.trial123:> grep trial123.front.sepia.ceph.com /etc/hosts
2026-02-12T11:07:24.660 INFO:teuthology.orchestra.run.trial123.stdout:10.20.193.123 trial123.front.sepia.ceph.com trial123
2026-02-12T11:07:24.661 DEBUG:teuthology.orchestra.run.trial123:> sudo hostname trial123
2026-02-12T11:07:24.729 DEBUG:teuthology.orchestra.run.trial123:> sudo sed -i -e 's/trial123.front.sepia.ceph.com/trial123/g' /etc/hosts
2026-02-12T11:07:24.795 DEBUG:teuthology.orchestra.run.trial123:> sudo sed -i -e 's/10.20.193.123/10.20.193.123/g' /etc/hosts
2026-02-12T11:07:24.861 DEBUG:teuthology.orchestra.run.trial123:> sudo sed -i -e 's/trial123.front.sepia.ceph.com/trial123/g' /etc/hostname
2026-02-12T11:07:24.925 DEBUG:teuthology.orchestra.run.trial123:> sudo hostname trial123
2026-02-12T11:07:24.989 DEBUG:teuthology.orchestra.run.trial123:> cat /etc/os-release
2026-02-12T11:07:25.045 INFO:teuthology.orchestra.run.trial123.stdout:NAME="CentOS Stream"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:VERSION="9"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:ID="centos"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:ID_LIKE="rhel fedora"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:VERSION_ID="9"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:PLATFORM_ID="platform:el9"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:ANSI_COLOR="0;31"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:LOGO="fedora-logo-icon"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T11:07:25.046 INFO:teuthology.orchestra.run.trial123.stdout:HOME_URL="https://centos.org/"
2026-02-12T11:07:25.047 INFO:teuthology.orchestra.run.trial123.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T11:07:25.047 INFO:teuthology.orchestra.run.trial123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T11:07:25.047 INFO:teuthology.orchestra.run.trial123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T11:07:25.047 INFO:teuthology.provision.fog.trial123:Deploy complete!
2026-02-12T11:07:25.048 DEBUG:teuthology.parallel:result is None
2026-02-12T11:07:28.425 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-12T11:07:28.425 DEBUG:teuthology.orchestra.run.trial043:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-12T11:07:28.578 INFO:teuthology.provision.fog.trial043:Node is ready
2026-02-12T11:07:28.578 DEBUG:teuthology.orchestra.run.trial043:> hostname
2026-02-12T11:07:28.633 INFO:teuthology.orchestra.run.trial043.stdout:trial043.front.sepia.ceph.com
2026-02-12T11:07:28.634 DEBUG:teuthology.orchestra.run.trial043:> grep trial043.front.sepia.ceph.com /etc/hosts
2026-02-12T11:07:28.689 INFO:teuthology.orchestra.run.trial043.stdout:10.20.193.43 trial043.front.sepia.ceph.com trial043
2026-02-12T11:07:28.690 DEBUG:teuthology.orchestra.run.trial043:> sudo hostname trial043
2026-02-12T11:07:28.758 DEBUG:teuthology.orchestra.run.trial043:> sudo sed -i -e 's/trial043.front.sepia.ceph.com/trial043/g' /etc/hosts
2026-02-12T11:07:28.824 DEBUG:teuthology.orchestra.run.trial043:> sudo sed -i -e 's/10.20.193.43/10.20.193.43/g' /etc/hosts
2026-02-12T11:07:28.889 DEBUG:teuthology.orchestra.run.trial043:> sudo sed -i -e 's/trial043.front.sepia.ceph.com/trial043/g' /etc/hostname
2026-02-12T11:07:28.952 DEBUG:teuthology.orchestra.run.trial043:> sudo hostname trial043
2026-02-12T11:07:29.016 DEBUG:teuthology.orchestra.run.trial043:> cat /etc/os-release
2026-02-12T11:07:29.072 INFO:teuthology.orchestra.run.trial043.stdout:NAME="CentOS Stream"
2026-02-12T11:07:29.072 INFO:teuthology.orchestra.run.trial043.stdout:VERSION="9"
2026-02-12T11:07:29.072 INFO:teuthology.orchestra.run.trial043.stdout:ID="centos"
2026-02-12T11:07:29.072 INFO:teuthology.orchestra.run.trial043.stdout:ID_LIKE="rhel fedora"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:VERSION_ID="9"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:PLATFORM_ID="platform:el9"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:ANSI_COLOR="0;31"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:LOGO="fedora-logo-icon"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:HOME_URL="https://centos.org/"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T11:07:29.073 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T11:07:29.074 INFO:teuthology.provision.fog.trial043:Deploy complete!
2026-02-12T11:07:29.075 DEBUG:teuthology.parallel:result is None
2026-02-12T11:07:29.456 INFO:teuthology.lock.ops:Checking trial043.front.sepia.ceph.com
2026-02-12T11:07:29.457 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-12T11:07:29.517 INFO:teuthology.lock.ops:Checking trial123.front.sepia.ceph.com
2026-02-12T11:07:29.517 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-12T11:07:29.576 INFO:teuthology.lock.ops:Updating [trial043.front.sepia.ceph.com]: set os type and version on server
2026-02-12T11:07:29.576 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial043.front.sepia.ceph.com'
2026-02-12T11:07:29.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial043.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:07:29.638 DEBUG:teuthology.orchestra.run.trial043:> true
2026-02-12T11:07:29.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial043.front.sepia.ceph.com'
2026-02-12T11:07:29.710 DEBUG:teuthology.orchestra.run.trial043:> uname -m
2026-02-12T11:07:29.766 INFO:teuthology.orchestra.run.trial043.stdout:x86_64
2026-02-12T11:07:29.766 DEBUG:teuthology.orchestra.run.trial043:> cat /etc/os-release
2026-02-12T11:07:29.822 INFO:teuthology.orchestra.run.trial043.stdout:NAME="CentOS Stream"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:VERSION="9"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:ID="centos"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:ID_LIKE="rhel fedora"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:VERSION_ID="9"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:PLATFORM_ID="platform:el9"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:ANSI_COLOR="0;31"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:LOGO="fedora-logo-icon"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T11:07:29.823 INFO:teuthology.orchestra.run.trial043.stdout:HOME_URL="https://centos.org/"
2026-02-12T11:07:29.824 INFO:teuthology.orchestra.run.trial043.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T11:07:29.824 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T11:07:29.824 INFO:teuthology.orchestra.run.trial043.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T11:07:29.824 INFO:teuthology.lock.ops:Updating trial043.front.sepia.ceph.com on lock server
2026-02-12T11:07:29.929 INFO:teuthology.lock.ops:Updating [trial123.front.sepia.ceph.com]: set os type and version on server
2026-02-12T11:07:29.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial123.front.sepia.ceph.com'
2026-02-12T11:07:29.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-12T11:07:30.003 DEBUG:teuthology.orchestra.run.trial123:> true
2026-02-12T11:07:30.079 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial123.front.sepia.ceph.com'
2026-02-12T11:07:30.079 DEBUG:teuthology.orchestra.run.trial123:> uname -m
2026-02-12T11:07:30.135 INFO:teuthology.orchestra.run.trial123.stdout:x86_64
2026-02-12T11:07:30.135 DEBUG:teuthology.orchestra.run.trial123:> cat /etc/os-release
2026-02-12T11:07:30.191 INFO:teuthology.orchestra.run.trial123.stdout:NAME="CentOS Stream"
2026-02-12T11:07:30.191 INFO:teuthology.orchestra.run.trial123.stdout:VERSION="9"
2026-02-12T11:07:30.191 INFO:teuthology.orchestra.run.trial123.stdout:ID="centos"
2026-02-12T11:07:30.191 INFO:teuthology.orchestra.run.trial123.stdout:ID_LIKE="rhel fedora"
2026-02-12T11:07:30.191 INFO:teuthology.orchestra.run.trial123.stdout:VERSION_ID="9"
2026-02-12T11:07:30.191 INFO:teuthology.orchestra.run.trial123.stdout:PLATFORM_ID="platform:el9"
2026-02-12T11:07:30.191 INFO:teuthology.orchestra.run.trial123.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-12T11:07:30.192 INFO:teuthology.orchestra.run.trial123.stdout:ANSI_COLOR="0;31"
2026-02-12T11:07:30.192 INFO:teuthology.orchestra.run.trial123.stdout:LOGO="fedora-logo-icon"
2026-02-12T11:07:30.192 INFO:teuthology.orchestra.run.trial123.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-12T11:07:30.192 INFO:teuthology.orchestra.run.trial123.stdout:HOME_URL="https://centos.org/"
2026-02-12T11:07:30.192 INFO:teuthology.orchestra.run.trial123.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-12T11:07:30.192 INFO:teuthology.orchestra.run.trial123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-12T11:07:30.192 INFO:teuthology.orchestra.run.trial123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-12T11:07:30.193 INFO:teuthology.lock.ops:Updating trial123.front.sepia.ceph.com on lock server
2026-02-12T11:07:30.296 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:07:30.393 INFO:teuthology.dispatcher.supervisor:Running job 46154
2026-02-12T11:07:30.393 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/bin/teuthology -v --owner scheduled_amathuri@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/amathuri-2026-02-12_10:22:28-crimson-rados:thrash-wip-amat-fix-74504-distro-debug-trial/46154 --name amathuri-2026-02-12_10:22:28-crimson-rados:thrash-wip-amat-fix-74504-distro-debug-trial --description crimson-rados:thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/crimson_bluestore thrashers/{pggrow} thrashosds-health workloads/radosbench-high-concurrency} -- /home/teuthworker/mnt/teuthology/amathuri-2026-02-12_10:22:28-crimson-rados:thrash-wip-amat-fix-74504-distro-debug-trial/46154/orig.config.yaml
2026-02-12T11:07:30.393 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/amathuri-2026-02-12_10:22:28-crimson-rados:thrash-wip-amat-fix-74504-distro-debug-trial/46154
2026-02-12T11:07:30.393 INFO:teuthology.dispatcher.supervisor:Job PID: 3739616
2026-02-12T11:07:30.394 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-12T11:09:30.473 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:11:30.655 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:13:30.842 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:15:31.042 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:17:31.239 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:19:31.356 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:21:31.596 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:23:31.783 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:25:31.937 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:27:32.109 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:29:32.314 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:31:32.504 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:33:32.704 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:35:32.884 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:37:32.998 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:39:33.204 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:41:33.397 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:43:33.549 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:45:33.715 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:47:33.939 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:49:34.154 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:51:34.362 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:53:34.509 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:55:34.719 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:57:34.928 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T11:59:35.144 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:01:35.307 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:03:35.503 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:05:35.707 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:07:35.899 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:09:36.087 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:11:36.276 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:13:36.430 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:15:36.578 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:17:36.785 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:19:36.904 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:21:37.064 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:23:37.264 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:25:37.474 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:27:37.674 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:29:37.877 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-12T12:29:37.987 ERROR:teuthology.dispatcher.supervisor:Child exited with code 1
2026-02-12T12:29:38.108 INFO:teuthology.dispatcher.supervisor:Unlocking machines...
2026-02-12T12:29:38.616 INFO:teuthology.orchestra.console.trial043:Power off
2026-02-12T12:29:38.617 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T12:29:38.626 INFO:teuthology.orchestra.console.trial123:Power off
2026-02-12T12:29:38.626 DEBUG:teuthology.orchestra.console.trial123:pexpect command: ipmitool -H trial123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-12T12:29:38.654 DEBUG:teuthology.orchestra.console.trial043:power off output: Chassis Power Control: Down/Off
2026-02-12T12:29:38.662 DEBUG:teuthology.orchestra.console.trial123:power off output: Chassis Power Control: Down/Off
2026-02-12T12:29:42.659 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T12:29:42.665 DEBUG:teuthology.orchestra.console.trial123:pexpect command: ipmitool -H trial123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T12:29:42.768 DEBUG:teuthology.orchestra.console.trial043:check power output: Chassis Power is on
2026-02-12T12:29:42.772 DEBUG:teuthology.orchestra.console.trial123:check power output: Chassis Power is on
2026-02-12T12:29:46.773 DEBUG:teuthology.orchestra.console.trial043:pexpect command: ipmitool -H trial043.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T12:29:46.780 DEBUG:teuthology.orchestra.console.trial123:pexpect command: ipmitool -H trial123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-12T12:29:46.883 DEBUG:teuthology.orchestra.console.trial043:check power output: Chassis Power is off
2026-02-12T12:29:46.883 INFO:teuthology.orchestra.console.trial043:Power off completed
2026-02-12T12:29:46.890 DEBUG:teuthology.orchestra.console.trial123:check power output: Chassis Power is off
2026-02-12T12:29:46.890 INFO:teuthology.orchestra.console.trial123:Power off completed
2026-02-12T12:29:47.128 INFO:teuthology.lock.ops:unlocked: trial043.front.sepia.ceph.com
2026-02-12T12:29:47.141 INFO:teuthology.lock.ops:unlocked: trial123.front.sepia.ceph.com