2022-03-19T00:40:57.376 INFO:root:teuthology version: 1.1.0-a4e18662
2022-03-19T00:40:57.377 INFO:teuthology.results:seed: '819'
2022-03-19T00:48:45.740 INFO:root:teuthology version: 1.1.0-a4e18662
2022-03-19T00:48:45.747 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-19T01:57:50.507 INFO:teuthology.results:Tests finished! gathering results...
2022-03-19T01:57:51.232 INFO:scrape:Found 17 jobs
2022-03-19T01:57:56.596 INFO:scrape:Found 4 distinct failure reasons
2022-03-19T01:57:56.598 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-03-19T01:57:56.598 INFO:scrape:6 jobs: ['6745425', '6745415', '6745424', '6745422', '6745414', '6745417']
2022-03-19T01:57:56.599 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-19T01:57:56.599 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-19T01:57:56.600 INFO:scrape:
2022-03-19T01:57:56.600 INFO:scrape:Crash: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi105 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-03-19T01:57:56.601 INFO:scrape:ceph version 17.0.0-11194-g90ff54e5 (90ff54e5cb6947d9c802f98c889a9630288a1008) quincy (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f487e2213c0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f487e782844]
 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9ee) [0x7f487e7829ee]
 6: (rbd::mirror::NamespaceReplayer<librbd::ImageCtx>::init_remote_pool_watcher(rbd::mirror::Peer<librbd::ImageCtx>&, Context*)+0x3f9) [0x560e20282a19]
 7: (rbd::mirror::NamespaceReplayer<librbd::ImageCtx>::init_remote_pool_watchers(Context*)+0x1e0) [0x560e20287630]
 8: (rbd::mirror::NamespaceReplayer<librbd::ImageCtx>::handle_init_local_pool_watcher(int, Context*)+0x51) [0x560e20287d31]
 9: (Context::complete(int)+0xd) [0x560e202122ad]
 10: (boost::asio::detail::completion_handler<librbd::asio::ContextWQ::queue(Context*, int)::{lambda()#1}, boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x560e2021274a]
 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x560e203a2ed5]
 12: /lib/librados.so.2(+0xdf238) [0x7f487f487238]
 13: /lib/librados.so.2(+0xe4eda) [0x7f487f48ceda]
 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f487e0fade4]
 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f487e215609]
 16: clone()
2022-03-19T01:57:56.601 INFO:scrape:3 jobs: ['6745428', '6745419', '6745420']
2022-03-19T01:57:56.602 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-19T01:57:56.602 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-19T01:57:56.602 INFO:scrape:
2022-03-19T01:57:56.603 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi065 with status 30: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
2022-03-19T01:57:56.603 INFO:scrape:7 jobs: ['6745421', '6745427', '6745426', '6745413', '6745423', '6745416', '6745418']
2022-03-19T01:57:56.604 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-19T01:57:56.604 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-19T01:57:56.605 INFO:scrape:
2022-03-19T01:57:56.605 INFO:scrape:Failure: "2022-03-19T00:58:49.318770+0000 mon.a (mon.0) 91 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2022-03-19T01:57:56.606 INFO:scrape:1 jobs: ['6745412']
2022-03-19T01:57:56.606 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-19T01:57:56.607 INFO:scrape:
2022-03-19T01:57:56.608 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
failed:  17
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6745412]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:19:30
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745412/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745412/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a476cccc7e904ebe936101fe7225a4fe

    "2022-03-19T00:58:49.318770+0000 mon.a (mon.0) 91 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6745413]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:32:19
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745413/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745413/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f02e174368f241d795c618b834876d96

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi110 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6745414]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:27:04
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745414/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745414/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e3cca53023b14ce3a2b18945d3c0e54a

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi155 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745415]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:27:11
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745415/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745415/
sentry: https://sentry.ceph.com/organizations/ceph/?query=930a02f715164cada8b5f58e7b972eff

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi135 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745416]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:30:30
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745416/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745416/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ca3af50063bd4db99d6dab0e412b70ec

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi165 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6745417]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:55:33
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745417/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745417/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7d7e42db2303466cae494378accca472

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi171 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745418]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:33:54
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745418/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745418/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4eb4709d7b7d4dd999ca8af7694d15c0

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi107 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6745419]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   01:02:47
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745419/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745419/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f55e387eda854b3a82dcc1d5a3de7acf

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi138 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745420]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:55:45
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745420/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745420/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd3482b91e394ad09bb382bb4ea817ef

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi085 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745421]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:33:04
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745421/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745421/
sentry: https://sentry.ceph.com/organizations/ceph/?query=41b5e273e51c4c95963125777786779a

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi065 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6745422]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   01:01:24
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745422/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745422/
sentry: https://sentry.ceph.com/organizations/ceph/?query=366b3555ce5c4354b6124c2d52bb8f5d

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi081 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745423]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:30:37
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745423/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745423/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c871e0b1cfb141b7a10e32c990e72ec9

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi189 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6745424]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:53:47
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745424/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745424/
sentry: https://sentry.ceph.com/organizations/ceph/?query=52864d28aa54430a93a9a941066af899

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi146 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745425]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:30:33
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745425/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745425/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b29f004b1b874239b7d024da091cca5a

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi125 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6745426]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:34:09
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745426/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745426/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cc02786ae4c54676bf7406bc4b1aba5d

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi183 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6745427]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:32:59
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745427/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745427/
sentry: https://sentry.ceph.com/organizations/ceph/?query=acfc3af45baa43e7a38693ddd5367535

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi160 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6745428]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:52:45
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745428/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-18_15:32:19-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6745428/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f2c7cb2cb9e74ece8de7a0eed8883942

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi105 with
    status 30: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=90ff54e5cb6947d9c802f98c889a9630288a1008
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'