2024-03-11T09:56:09.113 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-11T09:56:09.114 INFO:teuthology.results:subset: '81/99'
2024-03-11T09:56:09.114 INFO:teuthology.results:seed: '5633'
2024-03-11T09:56:09.510 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-03-11T09:56:09.510 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-03-11T14:46:30.095 INFO:teuthology.results:Tests finished! gathering results...
2024-03-11T14:46:30.235 INFO:scrape:Found 1 jobs
2024-03-11T14:46:30.455 INFO:scrape:Found 1 distinct failure reasons
2024-03-11T14:46:30.455 INFO:scrape:Crash: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi165 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a4679433626e24f222f64f757c6ab31760167603 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh'
2024-03-11T14:46:30.455 INFO:scrape:ceph version 19.0.0-1979-gb117a47a (b117a47aa0ad81d207fdd9ba852068f2f262fea8) squid (dev)
 1: /lib64/libc.so.6(+0x54db0) [0x7f6cd3854db0]
 2: /lib64/libc.so.6(+0xa154c) [0x7f6cd38a154c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x187) [0x55e562fa6564]
 6: (KernelDevice::_aio_thread()+0x11e0) [0x55e563977550]
 7: ceph-osd(+0xdc66c1) [0x55e5639776c1]
 8: /lib64/libc.so.6(+0x9f802) [0x7f6cd389f802]
 9: /lib64/libc.so.6(+0x3f450) [0x7f6cd383f450]
2024-03-11T14:46:30.455 INFO:scrape:1 jobs: ['7590830']
2024-03-11T14:46:30.456 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'conf/{disable-pool-app}', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}']
2024-03-11T14:46:30.456 INFO:scrape:
2024-03-11T14:46:30.456 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2024-03-11_09:56:02-rbd-wip-dis-testing-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/dis-2024-03-11_09:56:02-rbd-wip-dis-testing-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/dis-2024-03-11_09:56:02-rbd-wip-dis-testing-distro-default-smithi/
failed:  1
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7590830]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}
-----------------------------------------------------------------
time:   04:39:52
info:   https://pulpito.ceph.com/dis-2024-03-11_09:56:02-rbd-wip-dis-testing-distro-default-smithi/7590830/
log:    http://qa-proxy.ceph.com/teuthology/dis-2024-03-11_09:56:02-rbd-wip-dis-testing-distro-default-smithi/7590830/
sentry: https://sentry.ceph.com/organizations/ceph/?query=258acf7bf7174169af70b7d694f1b1e3

    Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi165 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a4679433626e24f222f64f757c6ab31760167603
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh'