Found 41 jobs Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-14_13:57:58-rbd-wip-yuri2-testing-2023-07-13-1236-distro-default-smithi/7337727/teuthology.log Found 6 distinct failure reasons Timeout 3h running clone.client.0/qa/workunits/rbd/diff_continuous.sh 2 jobs: ['7337772', '7337733'] suites intersection: ['cluster/{fixed-3', 'msgr-failures/few', 'openstack}', 'rbd/nbd/{base/install', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] suites union: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{rhel_8}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_nbd_diff_continuous}'] Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e3289f744b55c911f5d9b695cc2dfaa7044d8c97 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 5 jobs: ['7337775', '7337721', '7337778', '7337739', '7337692'] suites intersection: ['openstack}'] suites union: ['centos_latest', 'clusters/{fixed-1', 'clusters/{fixed-3', 'config/copy-on-read', 'min-compat-client/octopus', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'pool/replicated-data-pool', 'rbd/librbd/{cache/writeback', 'rbd/valgrind/{base/install', 'supported-random-distro$/{centos_latest}', 'validator/memcheck', 'workloads/python_api_tests_with_defaults}', 'workloads/python_api_tests_with_journaling}', 'workloads/python_api_tests}'] Dead: None 6 jobs: ['7337727', '7337730', '7337701', '7337724', '7337718', '7337742'] Dead: b"2023-07-15T08:18:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ rbd --cluster cluster1 -p mirror --namespace '' ls" ceph version 18.0.0-4926-ge3289f74 (e3289f744b55c911f5d9b695cc2dfaa7044d8c97) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fb881d25cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55ac8fdf1739] 5: (KernelDevice::_aio_thread()+0x1686) [0x55ac90a10136] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55ac90a15bf1] 7: /lib64/libpthread.so.0(+0x81ca) [0x7fb881d1b1ca] 8: clone() 1 jobs: ['7337781'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] Failure: Command failed (workunit test rbd/test_librbd.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e3289f744b55c911f5d9b695cc2dfaa7044d8c97 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 VALGRIND=\'--tool=memcheck --leak-check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' 3 jobs: ['7337763', '7337760', '7337766'] suites intersection: ['centos_latest', 'clusters/{fixed-1', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck'] suites union: ['centos_latest', 'clusters/{fixed-1', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck', 'workloads/c_api_tests_with_defaults}', 'workloads/c_api_tests_with_journaling}', 'workloads/c_api_tests}'] Failure: Command failed (workunit test rbd/test_rbd_mirror.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e3289f744b55c911f5d9b695cc2dfaa7044d8c97 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 VALGRIND=\'--tool=memcheck --leak-check=full\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_rbd_mirror.sh' 1 jobs: ['7337749'] suites: ['centos_latest', 'clusters/{fixed-1', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/valgrind/{base/install', 'validator/memcheck', 'workloads/rbd_mirror}']