2022-03-17T21:47:12.428 INFO:root:teuthology version: 1.1.0-cd8299ff
2022-03-17T21:47:12.431 INFO:teuthology.results:seed: '7937'
2022-03-17T21:58:27.714 INFO:root:teuthology version: 1.1.0-cd8299ff
2022-03-17T21:58:27.750 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-17T23:05:32.594 INFO:teuthology.results:Tests finished! gathering results...
2022-03-17T23:05:33.172 INFO:scrape:Found 13 jobs
2022-03-17T23:05:39.907 INFO:scrape:Found 3 distinct failure reasons
2022-03-17T23:05:39.908 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
2022-03-17T23:05:39.909 INFO:scrape:3 jobs: ['6742807', '6742798', '6742803']
2022-03-17T23:05:39.909 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-17T23:05:39.910 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-17T23:05:39.910 INFO:scrape:
2022-03-17T23:05:39.911 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi187 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-03-17T23:05:39.911 INFO:scrape:6 jobs: ['6742808', '6742797', '6742796', '6742801', '6742802', '6742806']
2022-03-17T23:05:39.912 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-17T23:05:39.912 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-17T23:05:39.912 INFO:scrape:
2022-03-17T23:05:39.913 INFO:scrape:Crash: Found coredumps on ubuntu@smithi125.front.sepia.ceph.com
2022-03-17T23:05:39.913 INFO:scrape:ceph version 17.0.0-11156-gbd53a04b (bd53a04b58fa92a82a5f1702df59a11febbfceae) quincy (dev)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f73f8037c20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f73f8c05262]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x284425) [0x7f73f8c05425]
 6: (rbd::mirror::NamespaceReplayer<librbd::ImageCtx>::init_remote_pool_watcher(rbd::mirror::Peer<librbd::ImageCtx>&, Context*)+0x40c) [0x560c214b9e9c]
 7: (rbd::mirror::NamespaceReplayer<librbd::ImageCtx>::init_remote_pool_watchers(Context*)+0x299) [0x560c214bd949]
 8: (rbd::mirror::NamespaceReplayer<librbd::ImageCtx>::handle_init_local_pool_watcher(int, Context*)+0x51) [0x560c214bdf01]
 9: (Context::complete(int)+0xd) [0x560c21448f6d]
 10: (boost::asio::detail::completion_handler<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x560c214493fa]
 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x560c21617765]
 12: /lib64/librados.so.2(+0xc8ee8) [0x7f73fa16eee8]
 13: /lib64/librados.so.2(+0xce28a) [0x7f73fa17428a]
 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f73f6e46ba3]
 15: /lib64/libpthread.so.0(+0x817f) [0x7f73f802d17f]
 16: clone()
2022-03-17T23:05:39.914 INFO:scrape:4 jobs: ['6742805', '6742804', '6742799', '6742800']
2022-03-17T23:05:39.914 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-17T23:05:39.915 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-17T23:05:39.915 INFO:scrape:
2022-03-17T23:05:39.916 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
failed:  13
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6742796]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:49:09
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742796/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742796/
sentry: https://sentry.ceph.com/organizations/ceph/?query=619721274b9b4babaa68a1c007da4378

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi201 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6742797]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:30:14
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742797/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742797/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5a046a7a81844dcda7d60d07f2a89c2d

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi135 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6742798]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:35:16
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742798/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742798/
sentry: https://sentry.ceph.com/organizations/ceph/?query=53df6082fa164d978ab5a362cd20715b

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi165 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6742799]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:51:50
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742799/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742799/

    Found coredumps on ubuntu@smithi145.front.sepia.ceph.com

[6742800]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   01:02:33
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742800/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742800/

    Found coredumps on ubuntu@smithi195.front.sepia.ceph.com

[6742801]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:35:00
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742801/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742801/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ffd882a2388b4e9fa0be75d312ff76ba

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi156 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6742802]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:27:38
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742802/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742802/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e349b1ff321342dca551ee5f4c7f2764

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi111 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6742803]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:34:57
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742803/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742803/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0d8170d6ce8246ff92edb321d4390331

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi152 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6742804]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:49:38
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742804/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742804/

    Found coredumps on ubuntu@smithi133.front.sepia.ceph.com

[6742805]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:55:08
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742805/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742805/

    Found coredumps on ubuntu@smithi125.front.sepia.ceph.com

[6742806]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:36:33
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742806/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742806/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1d780d888c30443fbb48e62e7c7e2418

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi173 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6742807]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:31:57
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742807/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742807/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9efd1031221e417bb98c1e49de107d11

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi148 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6742808]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:27:31
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742808/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:30:41-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742808/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6bb5e6fcccfb4e0fb26fa5ecf2da457e

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi187 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'