2024-01-20T22:52:31.180 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-20T22:52:31.180 INFO:teuthology.results:subset: '111/12000' 2024-01-20T22:52:31.180 INFO:teuthology.results:seed: '154' 2024-01-20T23:27:03.387 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-20T23:27:03.390 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-21T00:35:08.458 INFO:teuthology.results:Tests finished! gathering results... 2024-01-21T00:35:09.108 INFO:scrape:Found 16 jobs 2024-01-21T00:35:09.122 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523262/teuthology.log 2024-01-21T00:35:10.197 INFO:scrape:Found 4 distinct failure reasons 2024-01-21T00:35:10.198 INFO:scrape:Failure: Command failed on smithi135 with status 1: 'sudo yum install -y kernel' 2024-01-21T00:35:10.198 INFO:scrape:1 jobs: ['7523267'] 2024-01-21T00:35:10.198 INFO:scrape:suites: ['ignorelist_health', 'objectstore/bluestore-comp-zstd', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-distros/centos_latest', 'tasks/readwrite', 'thrashosds-health}'] 2024-01-21T00:35:10.198 INFO:scrape: 2024-01-21T00:35:10.198 INFO:scrape:Dead: None 2024-01-21T00:35:10.198 INFO:scrape:2 jobs: ['7523262', '7523263'] 2024-01-21T00:35:10.198 INFO:scrape: 2024-01-21T00:35:10.198 INFO:scrape:Failure: SSH connection to smithi039 was lost: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' 2024-01-21T00:35:10.199 INFO:scrape:1 jobs: ['7523265'] 2024-01-21T00:35:10.199 INFO:scrape:suites: ['ignorelist_health', 'objectstore/bluestore-comp-snappy', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-distros/centos_latest', 'tasks/rados_api_tests', 'thrashosds-health}'] 2024-01-21T00:35:10.199 INFO:scrape: 2024-01-21T00:35:10.199 INFO:scrape:Failure: Command failed (workunit test suites/fsx.sh) on smithi045 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-01-21T00:35:10.199 INFO:scrape:1 jobs: ['7523261'] 2024-01-21T00:35:10.199 INFO:scrape:suites: ['ignorelist_health', 'objectstore/bluestore-low-osd-mem-target', 'powercycle/default', 'powercycle/osd/{clusters/3osd-1per-target', 'supported-distros/centos_latest', 'tasks/cfuse_workunit_suites_fsx', 'thrashosds-health}'] 2024-01-21T00:35:10.199 INFO:scrape: 2024-01-21T00:35:10.200 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/ failed: 2 dead: 3 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [7523261] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-low-osd-mem-target powercycle/default supported-distros/centos_latest tasks/cfuse_workunit_suites_fsx thrashosds-health} ----------------------------------------------------------------- time: 00:11:47 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523261/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523261/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f68b902527a74a0eace488546ab4e293 Command failed (workunit test suites/fsx.sh) on smithi045 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a00c9940991c5e1e9ab595faa3f6e1281d8ae167 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' [7523267] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zstd powercycle/default supported-distros/centos_latest tasks/readwrite thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523267/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523267/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa00498c6f6d4e7c847d3f779a7b75df Command failed on smithi135 with status 1: 'sudo yum install -y kernel' Dead ================================================================= [7523262] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-stupid powercycle/default supported-distros/ubuntu_latest tasks/cfuse_workunit_suites_fsync thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523262/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523262/ Error reimaging machines: Expected smithi135's OS to be ubuntu 22.04 but found centos 9 [7523263] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-bitmap powercycle/default supported-distros/centos_latest tasks/cfuse_workunit_suites_pjd thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523263/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523263/ Error reimaging machines: Failed to power on smithi039 [7523265] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-snappy powercycle/default supported-distros/centos_latest tasks/rados_api_tests thrashosds-health} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523265/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523265/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e7ffbcaf89640cb8a4c1ce840946c00 SSH connection to smithi039 was lost: 'sudo grub2-mkconfig -o /boot/grub2/grub.cfg' Pass ================================================================= [7523254] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-stupid powercycle/default supported-distros/ubuntu_latest tasks/snaps-few-objects thrashosds-health} time: 00:28:36 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523254/ [7523255] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-bitmap powercycle/default supported-distros/centos_latest tasks/snaps-many-objects thrashosds-health} time: 00:23:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523255/ [7523256] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-lz4 powercycle/default supported-distros/ubuntu_latest tasks/admin_socket_objecter_requests thrashosds-health} time: 00:15:52 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523256/ [7523257] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-snappy powercycle/default supported-distros/centos_latest tasks/cfuse_workunit_kernel_untar_build thrashosds-health} time: 00:39:34 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523257/ [7523258] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zlib powercycle/default supported-distros/ubuntu_latest tasks/cfuse_workunit_misc thrashosds-health} time: 00:35:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523258/ [7523259] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zstd powercycle/default supported-distros/centos_latest tasks/cfuse_workunit_suites_ffsb thrashosds-health} time: 00:26:27 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523259/ [7523260] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-hybrid powercycle/default supported-distros/ubuntu_latest tasks/cfuse_workunit_suites_fsstress thrashosds-health} time: 00:13:05 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523260/ [7523264] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-lz4 powercycle/default supported-distros/ubuntu_latest tasks/cfuse_workunit_suites_truncate_delay thrashosds-health} time: 00:12:00 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523264/ [7523266] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-comp-zlib powercycle/default supported-distros/ubuntu_latest tasks/radosbench thrashosds-health} time: 01:06:02 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523266/ [7523268] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-hybrid powercycle/default supported-distros/ubuntu_latest tasks/snaps-few-objects thrashosds-health} time: 00:27:30 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523268/ [7523269] powercycle/osd/{clusters/3osd-1per-target ignorelist_health objectstore/bluestore-low-osd-mem-target powercycle/default supported-distros/centos_latest tasks/snaps-many-objects thrashosds-health} time: 00:26:04 info: https://pulpito.ceph.com/yuriw-2024-01-19_15:31:50-powercycle-main-distro-default-smithi/7523269/