2024-03-12T22:36:46.200 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-12T22:36:46.201 INFO:teuthology.results:subset: '111/120000' 2024-03-12T22:36:46.201 INFO:teuthology.results:seed: '8717' 2024-03-12T22:39:43.435 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-12T22:39:43.439 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-13T03:50:08.017 INFO:teuthology.results:Tests finished! gathering results... 2024-03-13T03:50:08.778 INFO:scrape:Found 9 jobs 2024-03-13T03:50:10.645 INFO:scrape:Found 3 distinct failure reasons 2024-03-13T03:50:10.645 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/cli_generic.sh 2024-03-13T03:50:10.645 INFO:scrape:1 jobs: ['7596019'] 2024-03-13T03:50:10.646 INFO:scrape:suites: ['clusters/{fixed-1', 'conf/{disable-pool-app}', 'features/format-1', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'pool/small-cache-pool', 'rbd/cli_v1/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd_cli_generic}'] 2024-03-13T03:50:10.646 INFO:scrape: 2024-03-13T03:50:10.646 INFO:scrape:Failure: Command failed (workunit test rbd/compare_mirror_images.sh) on smithi176 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50f1483466ba65f7424d0c0ee5ea54440fad33f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh' 2024-03-13T03:50:10.646 INFO:scrape:2 jobs: ['7596021', '7596020'] 2024-03-13T03:50:10.646 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install'] 2024-03-13T03:50:10.646 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'conf/{disable-pool-app}', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'workloads/compare-mirror-images-krbd}', 'workloads/compare-mirror-images-nbd}'] 2024-03-13T03:50:10.646 INFO:scrape: 2024-03-13T03:50:10.647 INFO:scrape:Crash: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi106 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50f1483466ba65f7424d0c0ee5ea54440fad33f8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2024-03-13T03:50:10.647 INFO:scrape:ceph version 17.2.7-632-g50f14834 (50f1483466ba65f7424d0c0ee5ea54440fad33f8) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55ce45ff4139] 2: (KernelDevice::_aio_thread()+0x1699) [0x55ce46cb0719] 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55ce46cb4501] 4: /lib64/libpthread.so.0(+0x81ca) [0x7f318506d1ca] 2024-03-13T03:50:10.647 INFO:scrape:1 jobs: ['7596024'] 2024-03-13T03:50:10.647 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}'] 2024-03-13T03:50:10.647 INFO:scrape: