2026-02-24T07:45:35.925 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0
2026-02-24T07:45:35.926 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:45:36.002 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:45:36.212 DEBUG:teuthology.exit:Installing handler: Handler(exiter=<teuthology.exit.Exiter object at 0x7fcefd595ac0>, func=<function ConsoleLog.start_logging.<locals>.kill_console_loggers at 0x7fcefcfbade0>, signals=[15])
2026-02-24T07:45:36.212 INFO:teuthology.lock.ops:Start node 'trial145.front.sepia.ceph.com' reimaging
2026-02-24T07:45:36.212 INFO:teuthology.lock.ops:Updating [trial145.front.sepia.ceph.com]: reset os type and version on server
2026-02-24T07:45:36.212 INFO:teuthology.lock.ops:Updating trial145.front.sepia.ceph.com on lock server
2026-02-24T07:45:36.272 INFO:teuthology.lock.ops:Start node 'trial179.front.sepia.ceph.com' reimaging
2026-02-24T07:45:36.272 INFO:teuthology.lock.ops:Updating [trial179.front.sepia.ceph.com]: reset os type and version on server
2026-02-24T07:45:36.272 INFO:teuthology.lock.ops:Updating trial179.front.sepia.ceph.com on lock server
2026-02-24T07:45:36.505 DEBUG:teuthology.provision.fog.trial145:Requesting image trial_centos_9.stream (ID 3)
2026-02-24T07:45:36.515 DEBUG:teuthology.provision.fog.trial179:Requesting image trial_centos_9.stream (ID 3)
2026-02-24T07:45:36.556 INFO:teuthology.provision.fog.trial145:Scheduling deploy of centos 9.stream
2026-02-24T07:45:36.567 INFO:teuthology.provision.fog.trial179:Scheduling deploy of centos 9.stream
2026-02-24T07:45:37.883 INFO:teuthology.orchestra.console.trial145:Power off
2026-02-24T07:45:37.883 DEBUG:teuthology.orchestra.console.trial145:pexpect command: ipmitool -H trial145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T07:45:37.902 INFO:teuthology.orchestra.console.trial179:Power off
2026-02-24T07:45:37.902 DEBUG:teuthology.orchestra.console.trial179:pexpect command: ipmitool -H trial179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T07:45:37.922 DEBUG:teuthology.orchestra.console.trial145:power off output: Chassis Power Control: Down/Off
2026-02-24T07:45:37.941 DEBUG:teuthology.orchestra.console.trial179:power off output: Chassis Power Control: Down/Off
2026-02-24T07:45:41.926 DEBUG:teuthology.orchestra.console.trial145:pexpect command: ipmitool -H trial145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:45:41.942 DEBUG:teuthology.orchestra.console.trial179:pexpect command: ipmitool -H trial179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:45:42.036 DEBUG:teuthology.orchestra.console.trial145:check power output: Chassis Power is off
2026-02-24T07:45:42.036 INFO:teuthology.orchestra.console.trial145:Power off completed
2026-02-24T07:45:42.050 DEBUG:teuthology.orchestra.console.trial179:check power output: Chassis Power is off
2026-02-24T07:45:42.050 INFO:teuthology.orchestra.console.trial179:Power off completed
2026-02-24T07:45:42.138 INFO:teuthology.orchestra.console.trial145:Power on
2026-02-24T07:45:42.138 DEBUG:teuthology.orchestra.console.trial145:pexpect command: ipmitool -H trial145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-24T07:45:42.151 INFO:teuthology.orchestra.console.trial179:Power on
2026-02-24T07:45:42.151 DEBUG:teuthology.orchestra.console.trial179:pexpect command: ipmitool -H trial179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on
2026-02-24T07:45:42.182 DEBUG:teuthology.orchestra.console.trial145:power on output: Chassis Power Control: Up/On
2026-02-24T07:45:42.192 DEBUG:teuthology.orchestra.console.trial179:power on output: Chassis Power Control: Up/On
2026-02-24T07:45:46.187 DEBUG:teuthology.orchestra.console.trial145:pexpect command: ipmitool -H trial145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:45:46.196 DEBUG:teuthology.orchestra.console.trial179:pexpect command: ipmitool -H trial179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T07:45:46.300 DEBUG:teuthology.orchestra.console.trial145:check power output: Chassis Power is on
2026-02-24T07:45:46.300 INFO:teuthology.orchestra.console.trial145:Power on completed
2026-02-24T07:45:46.307 DEBUG:teuthology.orchestra.console.trial179:check power output: Chassis Power is on
2026-02-24T07:45:46.307 INFO:teuthology.orchestra.console.trial179:Power on completed
2026-02-24T07:45:46.402 INFO:teuthology.provision.fog.trial145:Waiting for deploy to finish
2026-02-24T07:45:46.407 INFO:teuthology.provision.fog.trial179:Waiting for deploy to finish
2026-02-24T07:47:34.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:47:34.370 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:47:37.472 WARNING:teuthology.provision.fog.trial179:[Errno None] Unable to connect to port 22 on 10.20.193.179
2026-02-24T07:47:43.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:47:46.560 WARNING:teuthology.provision.fog.trial179:[Errno None] Unable to connect to port 22 on 10.20.193.179
2026-02-24T07:47:52.567 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:47:55.648 WARNING:teuthology.provision.fog.trial179:[Errno None] Unable to connect to port 22 on 10.20.193.179
2026-02-24T07:47:56.927 WARNING:teuthology.provision.fog.trial145:[Errno None] Unable to connect to port 22 on 10.20.193.145
2026-02-24T07:48:01.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:02.929 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:04.735 WARNING:teuthology.provision.fog.trial179:[Errno None] Unable to connect to port 22 on 10.20.193.179
2026-02-24T07:48:06.016 WARNING:teuthology.provision.fog.trial145:[Errno None] Unable to connect to port 22 on 10.20.193.145
2026-02-24T07:48:10.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:12.017 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:13.823 WARNING:teuthology.provision.fog.trial179:[Errno None] Unable to connect to port 22 on 10.20.193.179
2026-02-24T07:48:15.104 WARNING:teuthology.provision.fog.trial145:[Errno None] Unable to connect to port 22 on 10.20.193.145
2026-02-24T07:48:19.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:21.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:55.931 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-24T07:48:55.932 DEBUG:teuthology.orchestra.run.trial179:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-24T07:48:56.167 INFO:teuthology.provision.fog.trial179:Node is ready
2026-02-24T07:48:56.168 DEBUG:teuthology.orchestra.run.trial179:> hostname
2026-02-24T07:48:56.224 INFO:teuthology.orchestra.run.trial179.stdout:trial179.front.sepia.ceph.com
2026-02-24T07:48:56.224 DEBUG:teuthology.orchestra.run.trial179:> grep trial179.front.sepia.ceph.com /etc/hosts
2026-02-24T07:48:56.280 INFO:teuthology.orchestra.run.trial179.stdout:10.20.193.179 trial179.front.sepia.ceph.com trial179
2026-02-24T07:48:56.280 DEBUG:teuthology.orchestra.run.trial179:> sudo hostname trial179
2026-02-24T07:48:56.352 DEBUG:teuthology.orchestra.run.trial179:> sudo sed -i -e 's/trial179.front.sepia.ceph.com/trial179/g' /etc/hosts
2026-02-24T07:48:56.419 DEBUG:teuthology.orchestra.run.trial179:> sudo sed -i -e 's/10.20.193.179/10.20.193.179/g' /etc/hosts
2026-02-24T07:48:56.485 DEBUG:teuthology.orchestra.run.trial179:> sudo sed -i -e 's/trial179.front.sepia.ceph.com/trial179/g' /etc/hostname
2026-02-24T07:48:56.551 DEBUG:teuthology.orchestra.run.trial179:> sudo hostname trial179
2026-02-24T07:48:56.615 DEBUG:teuthology.orchestra.run.trial179:> cat /etc/os-release
2026-02-24T07:48:56.672 INFO:teuthology.orchestra.run.trial179.stdout:NAME="CentOS Stream"
2026-02-24T07:48:56.672 INFO:teuthology.orchestra.run.trial179.stdout:VERSION="9"
2026-02-24T07:48:56.672 INFO:teuthology.orchestra.run.trial179.stdout:ID="centos"
2026-02-24T07:48:56.672 INFO:teuthology.orchestra.run.trial179.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:48:56.672 INFO:teuthology.orchestra.run.trial179.stdout:VERSION_ID="9"
2026-02-24T07:48:56.672 INFO:teuthology.orchestra.run.trial179.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:48:56.672 INFO:teuthology.orchestra.run.trial179.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:48:56.673 INFO:teuthology.orchestra.run.trial179.stdout:ANSI_COLOR="0;31"
2026-02-24T07:48:56.673 INFO:teuthology.orchestra.run.trial179.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:48:56.673 INFO:teuthology.orchestra.run.trial179.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:48:56.673 INFO:teuthology.orchestra.run.trial179.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:48:56.673 INFO:teuthology.orchestra.run.trial179.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:48:56.673 INFO:teuthology.orchestra.run.trial179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:48:56.673 INFO:teuthology.orchestra.run.trial179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:48:56.674 INFO:teuthology.provision.fog.trial179:Deploy complete!
2026-02-24T07:48:56.675 DEBUG:teuthology.parallel:result is None
2026-02-24T07:48:56.955 INFO:teuthology.orchestra.run:Running command with timeout 600
2026-02-24T07:48:56.956 DEBUG:teuthology.orchestra.run.trial145:> while [ ! -e '/.cephlab_hostname_set' ]; do sleep 5; done
2026-02-24T07:48:57.103 INFO:teuthology.provision.fog.trial145:Node is ready
2026-02-24T07:48:57.104 DEBUG:teuthology.orchestra.run.trial145:> hostname
2026-02-24T07:48:57.163 INFO:teuthology.orchestra.run.trial145.stdout:trial145.front.sepia.ceph.com
2026-02-24T07:48:57.163 DEBUG:teuthology.orchestra.run.trial145:> grep trial145.front.sepia.ceph.com /etc/hosts
2026-02-24T07:48:57.223 INFO:teuthology.orchestra.run.trial145.stdout:10.20.193.145 trial145.front.sepia.ceph.com trial145
2026-02-24T07:48:57.223 DEBUG:teuthology.orchestra.run.trial145:> sudo hostname trial145
2026-02-24T07:48:57.295 DEBUG:teuthology.orchestra.run.trial145:> sudo sed -i -e 's/trial145.front.sepia.ceph.com/trial145/g' /etc/hosts
2026-02-24T07:48:57.364 DEBUG:teuthology.orchestra.run.trial145:> sudo sed -i -e 's/10.20.193.145/10.20.193.145/g' /etc/hosts
2026-02-24T07:48:57.431 DEBUG:teuthology.orchestra.run.trial145:> sudo sed -i -e 's/trial145.front.sepia.ceph.com/trial145/g' /etc/hostname
2026-02-24T07:48:57.498 DEBUG:teuthology.orchestra.run.trial145:> sudo hostname trial145
2026-02-24T07:48:57.565 DEBUG:teuthology.orchestra.run.trial145:> cat /etc/os-release
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:NAME="CentOS Stream"
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:VERSION="9"
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:ID="centos"
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:VERSION_ID="9"
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:48:57.625 INFO:teuthology.orchestra.run.trial145.stdout:ANSI_COLOR="0;31"
2026-02-24T07:48:57.626 INFO:teuthology.orchestra.run.trial145.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:48:57.626 INFO:teuthology.orchestra.run.trial145.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:48:57.626 INFO:teuthology.orchestra.run.trial145.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:48:57.626 INFO:teuthology.orchestra.run.trial145.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:48:57.626 INFO:teuthology.orchestra.run.trial145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:48:57.626 INFO:teuthology.orchestra.run.trial145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:48:57.626 INFO:teuthology.provision.fog.trial145:Deploy complete!
2026-02-24T07:48:57.627 DEBUG:teuthology.parallel:result is None
2026-02-24T07:48:58.025 INFO:teuthology.lock.ops:Checking trial145.front.sepia.ceph.com
2026-02-24T07:48:58.026 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-24T07:48:58.124 INFO:teuthology.lock.ops:Checking trial179.front.sepia.ceph.com
2026-02-24T07:48:58.124 INFO:teuthology.lock.ops:New key found. Updating...
2026-02-24T07:48:58.187 INFO:teuthology.lock.ops:Updating [trial145.front.sepia.ceph.com]: set os type and version on server
2026-02-24T07:48:58.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial145.front.sepia.ceph.com'
2026-02-24T07:48:58.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:58.249 DEBUG:teuthology.orchestra.run.trial145:> true
2026-02-24T07:48:58.322 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial145.front.sepia.ceph.com'
2026-02-24T07:48:58.322 DEBUG:teuthology.orchestra.run.trial145:> uname -m
2026-02-24T07:48:58.378 INFO:teuthology.orchestra.run.trial145.stdout:x86_64
2026-02-24T07:48:58.378 DEBUG:teuthology.orchestra.run.trial145:> cat /etc/os-release
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:NAME="CentOS Stream"
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:VERSION="9"
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:ID="centos"
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:VERSION_ID="9"
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:48:58.434 INFO:teuthology.orchestra.run.trial145.stdout:ANSI_COLOR="0;31"
2026-02-24T07:48:58.435 INFO:teuthology.orchestra.run.trial145.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:48:58.435 INFO:teuthology.orchestra.run.trial145.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:48:58.435 INFO:teuthology.orchestra.run.trial145.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:48:58.435 INFO:teuthology.orchestra.run.trial145.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:48:58.435 INFO:teuthology.orchestra.run.trial145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:48:58.435 INFO:teuthology.orchestra.run.trial145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:48:58.436 INFO:teuthology.lock.ops:Updating trial145.front.sepia.ceph.com on lock server
2026-02-24T07:48:58.537 INFO:teuthology.lock.ops:Updating [trial179.front.sepia.ceph.com]: set os type and version on server
2026-02-24T07:48:58.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial179.front.sepia.ceph.com'
2026-02-24T07:48:58.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T07:48:58.611 DEBUG:teuthology.orchestra.run.trial179:> true
2026-02-24T07:48:58.689 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial179.front.sepia.ceph.com'
2026-02-24T07:48:58.689 DEBUG:teuthology.orchestra.run.trial179:> uname -m
2026-02-24T07:48:58.745 INFO:teuthology.orchestra.run.trial179.stdout:x86_64
2026-02-24T07:48:58.745 DEBUG:teuthology.orchestra.run.trial179:> cat /etc/os-release
2026-02-24T07:48:58.801 INFO:teuthology.orchestra.run.trial179.stdout:NAME="CentOS Stream"
2026-02-24T07:48:58.801 INFO:teuthology.orchestra.run.trial179.stdout:VERSION="9"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:ID="centos"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:ID_LIKE="rhel fedora"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:VERSION_ID="9"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:PLATFORM_ID="platform:el9"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:PRETTY_NAME="CentOS Stream 9"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:ANSI_COLOR="0;31"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:LOGO="fedora-logo-icon"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:CPE_NAME="cpe:/o:centos:centos:9"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:HOME_URL="https://centos.org/"
2026-02-24T07:48:58.802 INFO:teuthology.orchestra.run.trial179.stdout:BUG_REPORT_URL="https://issues.redhat.com/"
2026-02-24T07:48:58.803 INFO:teuthology.orchestra.run.trial179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
2026-02-24T07:48:58.803 INFO:teuthology.orchestra.run.trial179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2026-02-24T07:48:58.803 INFO:teuthology.lock.ops:Updating trial179.front.sepia.ceph.com on lock server
2026-02-24T07:48:58.913 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:48:59.005 INFO:teuthology.dispatcher.supervisor:Running job 67613
2026-02-24T07:48:59.005 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/bin/teuthology -v --owner scheduled_shraddhaag@soko04.front.sepia.ceph.com --archive /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613 --name shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial --description crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/cephadm objectstore/crimson_bluestore thrashers/{default} thrashosds-health workloads/radosbench} -- /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613/orig.config.yaml
2026-02-24T07:48:59.006 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613
2026-02-24T07:48:59.006 INFO:teuthology.dispatcher.supervisor:Job PID: 2979422
2026-02-24T07:48:59.006 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2026-02-24T07:50:59.106 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:52:59.301 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:54:59.485 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:56:59.655 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T07:58:59.858 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:01:00.081 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:03:00.265 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:05:00.373 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:07:00.554 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:09:00.656 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:11:00.807 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:13:01.002 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:15:04.149 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:17:09.370 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:19:09.583 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:21:09.777 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:23:09.953 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:25:10.132 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:27:10.292 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:29:10.512 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:31:10.691 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:33:10.906 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:35:11.125 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:37:11.332 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:39:11.478 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:41:11.625 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:43:11.831 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:45:12.037 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:47:12.259 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:49:12.452 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:51:12.652 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:53:12.843 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:55:13.047 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:57:13.247 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T08:59:13.361 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:01:13.477 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:03:13.699 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:05:13.911 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:07:14.107 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:09:14.277 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:11:14.392 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:13:14.578 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:15:14.775 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:17:14.965 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:19:15.079 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:21:15.349 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:23:15.469 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:25:15.666 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:27:15.865 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:29:16.038 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:31:16.214 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:33:18.112 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:35:18.324 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:37:18.509 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:39:18.711 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:41:18.898 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:43:19.082 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:45:19.251 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:47:19.426 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:49:19.608 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:51:19.789 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:53:19.967 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:55:20.148 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:57:20.299 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T09:59:20.528 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:01:20.726 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:03:20.926 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:05:21.152 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:07:21.344 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:09:21.550 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:11:21.758 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:13:21.944 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:15:22.069 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:17:22.271 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:19:22.471 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:21:22.674 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:23:22.848 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:25:23.026 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:27:23.210 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:29:23.389 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:31:23.576 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:33:23.755 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:35:23.920 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:37:24.094 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:39:24.251 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:41:24.409 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:43:24.620 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:45:24.823 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:47:25.002 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:49:25.184 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:51:25.373 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:53:25.575 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:55:25.803 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:57:25.991 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T10:59:26.183 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:01:26.353 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:03:26.454 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:05:26.572 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:07:26.747 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:09:26.952 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:11:27.128 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:13:27.295 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:15:27.472 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:17:27.620 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:19:27.824 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:21:28.010 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:23:28.204 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:25:28.365 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:27:28.541 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:29:28.738 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:31:28.923 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:33:29.046 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:35:29.261 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:37:29.461 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:39:36.391 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:41:36.590 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:43:36.833 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:45:37.029 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:47:37.225 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:49:37.376 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:51:37.571 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:53:37.793 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:55:37.962 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:57:38.171 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T11:59:38.350 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:01:38.537 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:03:38.714 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:05:38.907 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:07:39.117 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:09:39.303 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:11:39.521 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:13:39.697 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:15:39.897 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:17:40.074 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:19:40.209 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:21:40.396 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:23:40.569 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:25:40.746 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:27:40.849 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:29:41.006 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:31:41.147 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:33:41.284 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:35:41.458 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:37:41.648 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:39:41.820 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:41:42.228 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:43:42.644 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:45:42.851 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:47:43.063 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:49:43.247 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:51:43.407 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:53:43.620 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:55:43.725 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:57:43.902 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T12:59:44.079 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:01:44.257 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:03:44.434 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:05:44.627 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:07:44.808 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:09:44.981 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:11:45.094 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:13:45.294 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:15:45.499 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:17:45.708 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:19:45.871 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:21:46.065 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:23:46.260 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:25:46.445 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:27:46.616 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:29:46.784 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:31:46.944 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:33:47.084 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:35:47.281 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:37:47.531 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:39:48.335 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:41:48.614 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:43:48.796 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:45:48.920 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:47:49.129 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:49:49.296 WARNING:teuthology.dispatcher.supervisor:Job shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613 ran longer than 21600s. Killing...
2026-02-24T13:49:49.380 INFO:teuthology.kill:Killing Pids: {2979422}
2026-02-24T13:49:49.381 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:49:49.495 INFO:teuthology.task.internal:roles: ubuntu@trial145.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a']
2026-02-24T13:49:49.495 INFO:teuthology.task.internal:roles: ubuntu@trial179.front.sepia.ceph.com - ['osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']
2026-02-24T13:49:49.495 INFO:teuthology.misc:Compressing logs...
2026-02-24T13:49:49.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial145.front.sepia.ceph.com'
2026-02-24T13:49:49.495 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T13:49:49.564 DEBUG:teuthology.orchestra.run.trial145:> true
2026-02-24T13:49:49.640 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial145.front.sepia.ceph.com'
2026-02-24T13:49:49.640 DEBUG:teuthology.orchestra.run.trial145:> sudo find /var/lib/ceph/crash -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:49:49.683 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial179.front.sepia.ceph.com'
2026-02-24T13:49:49.684 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60}
2026-02-24T13:49:49.784 DEBUG:teuthology.orchestra.run.trial179:> true
2026-02-24T13:49:49.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial179.front.sepia.ceph.com'
2026-02-24T13:49:49.861 DEBUG:teuthology.orchestra.run.trial179:> sudo find /var/lib/ceph/crash -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:49:49.932 DEBUG:teuthology.misc:Transferring archived files from trial145:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613/remote/trial145/crash
2026-02-24T13:49:49.933 DEBUG:teuthology.orchestra.run.trial145:> sudo tar c -f - -C /var/lib/ceph/crash -- .
2026-02-24T13:49:49.961 DEBUG:teuthology.misc:Transferring archived files from trial179:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613/remote/trial179/crash
2026-02-24T13:49:49.962 DEBUG:teuthology.orchestra.run.trial179:> sudo tar c -f - -C /var/lib/ceph/crash -- .
2026-02-24T13:49:49.994 INFO:teuthology.misc:Compressing logs...
2026-02-24T13:49:49.995 DEBUG:teuthology.orchestra.run.trial145:> sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:49:50.005 DEBUG:teuthology.orchestra.run.trial179:> sudo find /home/ubuntu/cephtest/archive -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:49:50.027 INFO:teuthology.orchestra.run.trial145.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2026-02-24T13:49:50.027 INFO:teuthology.orchestra.run.trial145.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2026-02-24T13:49:50.027 INFO:teuthology.orchestra.run.trial145.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2026-02-24T13:49:50.028 INFO:teuthology.orchestra.run.trial145.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2026-02-24T13:49:50.059 INFO:teuthology.orchestra.run.trial179.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2026-02-24T13:49:50.059 INFO:teuthology.orchestra.run.trial179.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2026-02-24T13:49:50.060 INFO:teuthology.orchestra.run.trial179.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2026-02-24T13:49:50.060 INFO:teuthology.orchestra.run.trial179.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2026-02-24T13:49:50.062 DEBUG:teuthology.misc:Transferring archived files from trial145:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613/remote/trial145/
2026-02-24T13:49:50.062 DEBUG:teuthology.orchestra.run.trial145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2026-02-24T13:49:50.096 DEBUG:teuthology.misc:Transferring archived files from trial179:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613/remote/trial179/
2026-02-24T13:49:50.096 DEBUG:teuthology.orchestra.run.trial179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- .
2026-02-24T13:49:50.128 INFO:teuthology.misc:Compressing logs...
2026-02-24T13:49:50.128 DEBUG:teuthology.orchestra.run.trial145:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:49:50.139 DEBUG:teuthology.orchestra.run.trial179:> sudo find /var/log/ceph -name *.log -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --
2026-02-24T13:49:50.162 INFO:teuthology.orchestra.run.trial145.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log
2026-02-24T13:49:50.162 INFO:teuthology.orchestra.run.trial145.stderr:gzip -5 --verbose -- /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mon.a.log
2026-02-24T13:49:50.162 INFO:teuthology.orchestra.run.trial145.stderr:/var/log/ceph/cephadm.log:	gzip -5 --verbose -- /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.cephadm.log
2026-02-24T13:49:50.162 INFO:teuthology.orchestra.run.trial145.stderr:gzip -5 --verbose -- /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.audit.log
2026-02-24T13:49:50.162 INFO:teuthology.orchestra.run.trial145.stderr:/var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mon.a.log:	/var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.cephadm.log:	gzip -5 --verbose -- /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-volume.log
2026-02-24T13:49:50.162 INFO:teuthology.orchestra.run.trial145.stderr: 82.0% -- replaced with /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.cephadm.log.gz
2026-02-24T13:49:50.163 INFO:teuthology.orchestra.run.trial145.stderr:/var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.audit.log:	gzip -5 --verbose -- /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mgr.x.log
2026-02-24T13:49:50.163 INFO:teuthology.orchestra.run.trial145.stderr:gzip -5 --verbose -- /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.log
2026-02-24T13:49:50.171 INFO:teuthology.orchestra.run.trial145.stderr:/var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-volume.log:	/var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mgr.x.log:	/var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.log:	 90.4% -- replaced with /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.audit.log.gz
2026-02-24T13:49:50.175 INFO:teuthology.orchestra.run.trial145.stderr: 93.4% -- replaced with /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph.log.gz
2026-02-24T13:49:50.176 INFO:teuthology.orchestra.run.trial145.stderr: 92.5% -- replaced with /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-volume.log.gz
2026-02-24T13:49:50.194 INFO:teuthology.orchestra.run.trial179.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log
2026-02-24T13:49:50.194 INFO:teuthology.orchestra.run.trial179.stderr:gzip -5 --verbose -- /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-volume.log
2026-02-24T13:49:50.196 INFO:teuthology.orchestra.run.trial179.stderr:/var/log/ceph/cephadm.log:	/var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-volume.log:	 87.3% -- replaced with /var/log/ceph/cephadm.log.gz
2026-02-24T13:49:50.203 INFO:teuthology.orchestra.run.trial145.stderr: 97.0% -- replaced with /var/log/ceph/cephadm.log.gz
2026-02-24T13:49:50.209 INFO:teuthology.orchestra.run.trial179.stderr: 92.5% -- replaced with /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-volume.log.gz
2026-02-24T13:49:51.088 INFO:teuthology.orchestra.run.trial145.stderr:gzip: /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mgr.x.log: file size changed while zipping
2026-02-24T13:49:51.088 INFO:teuthology.orchestra.run.trial145.stderr: 92.1% -- replaced with /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mgr.x.log.gz
2026-02-24T13:49:54.794 INFO:teuthology.orchestra.run.trial145.stderr:gzip: /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mon.a.log: file size changed while zipping
2026-02-24T13:49:54.794 INFO:teuthology.orchestra.run.trial145.stderr: 88.7% -- replaced with /var/log/ceph/d656f974-1156-11f1-8066-d404e6e7d460/ceph-mon.a.log.gz
2026-02-24T13:49:54.796 DEBUG:teuthology.misc:Transferring archived files from trial145:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613/remote/trial145/log
2026-02-24T13:49:54.797 DEBUG:teuthology.orchestra.run.trial145:> sudo tar c -f - -C /var/log/ceph -- .
2026-02-24T13:49:55.457 DEBUG:teuthology.misc:Transferring archived files from trial179:/var/log/ceph to /home/teuthworker/mnt/teuthology/shraddhaag-2026-02-24_06:51:01-crimson-rados-main-distro-debug-trial/67613/remote/trial179/log
2026-02-24T13:49:55.457 DEBUG:teuthology.orchestra.run.trial179:> sudo tar c -f - -C /var/log/ceph -- .
2026-02-24T13:49:55.568 INFO:teuthology.kill:No teuthology processes running
2026-02-24T13:49:55.569 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:49:56.285 INFO:teuthology.orchestra.console.trial179:Power off
2026-02-24T13:49:56.285 DEBUG:teuthology.orchestra.console.trial179:pexpect command: ipmitool -H trial179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T13:49:56.295 INFO:teuthology.orchestra.console.trial145:Power off
2026-02-24T13:49:56.295 DEBUG:teuthology.orchestra.console.trial145:pexpect command: ipmitool -H trial145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off
2026-02-24T13:49:56.329 DEBUG:teuthology.orchestra.console.trial179:power off output: Chassis Power Control: Down/Off
2026-02-24T13:49:56.333 DEBUG:teuthology.orchestra.console.trial145:power off output: Chassis Power Control: Down/Off
2026-02-24T13:50:00.333 DEBUG:teuthology.orchestra.console.trial179:pexpect command: ipmitool -H trial179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:50:00.340 DEBUG:teuthology.orchestra.console.trial145:pexpect command: ipmitool -H trial145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:50:00.443 DEBUG:teuthology.orchestra.console.trial179:check power output: Chassis Power is on
2026-02-24T13:50:00.450 DEBUG:teuthology.orchestra.console.trial145:check power output: Chassis Power is on
2026-02-24T13:50:04.447 DEBUG:teuthology.orchestra.console.trial179:pexpect command: ipmitool -H trial179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:50:04.453 DEBUG:teuthology.orchestra.console.trial145:pexpect command: ipmitool -H trial145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status
2026-02-24T13:50:04.556 DEBUG:teuthology.orchestra.console.trial179:check power output: Chassis Power is off
2026-02-24T13:50:04.556 INFO:teuthology.orchestra.console.trial179:Power off completed
2026-02-24T13:50:04.561 DEBUG:teuthology.orchestra.console.trial145:check power output: Chassis Power is off
2026-02-24T13:50:04.561 INFO:teuthology.orchestra.console.trial145:Power off completed
2026-02-24T13:50:04.811 INFO:teuthology.lock.ops:unlocked: trial179.front.sepia.ceph.com
2026-02-24T13:50:04.815 INFO:teuthology.lock.ops:unlocked: trial145.front.sepia.ceph.com
2026-02-24T13:50:04.816 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:52:04.929 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com
2026-02-24T13:52:05.025 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2026-02-24T13:52:05.144 WARNING:teuthology.dispatcher.supervisor:Was going to unlock trial145 but it was locked by another job: /home/teuthworker/mnt/teuthology/cjames-2026-02-24_10:01:51-rados-main-distro-default-trial/67855
2026-02-24T13:52:05.144 WARNING:teuthology.dispatcher.supervisor:Was going to unlock trial179 but it was locked by another job: /home/teuthworker/mnt/teuthology/cjames-2026-02-24_10:01:51-rados-main-distro-default-trial/67857