2022-03-11T17:13:44.150 INFO:root:teuthology version: 1.1.0-ff5db916
2022-03-11T17:13:44.151 INFO:teuthology.results:seed: '8973'
2022-03-11T17:44:41.080 INFO:root:teuthology version: 1.1.0-ff5db916
2022-03-11T17:44:41.085 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-11T18:30:44.558 INFO:teuthology.results:Tests finished! gathering results...
2022-03-11T18:30:45.982 INFO:scrape:Found 19 jobs
2022-03-11T18:30:53.079 INFO:scrape:Found 3 distinct failure reasons
2022-03-11T18:30:53.080 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-03-11T18:30:53.081 INFO:scrape:11 jobs: ['6732081', '6732076', '6732089', '6732085', '6732082', '6732077', '6732079', '6732072', '6732084', '6732075', '6732074']
2022-03-11T18:30:53.081 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-11T18:30:53.082 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-11T18:30:53.082 INFO:scrape:
2022-03-11T18:30:53.082 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
2022-03-11T18:30:53.083 INFO:scrape:7 jobs: ['6732083', '6732078', '6732088', '6732080', '6732073', '6732086', '6732090']
2022-03-11T18:30:53.083 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-11T18:30:53.084 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-11T18:30:53.084 INFO:scrape:
2022-03-11T18:30:53.085 INFO:scrape:Failure: "2022-03-11T17:59:42.345881+0000 mon.a (mon.0) 94 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2022-03-11T18:30:53.085 INFO:scrape:1 jobs: ['6732087']
2022-03-11T18:30:53.086 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-11T18:30:53.086 INFO:scrape:
2022-03-11T18:30:53.088 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
failed:  19
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6732072]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:19:29
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732072/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732072/
sentry: https://sentry.ceph.com/organizations/ceph/?query=442acda211524df6baaef6fd02c8541c

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi153 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732073]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:38:48
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732073/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732073/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd51f8a39284421cbc29cc26a9d094e9

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi163 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6732074]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:37:59
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732074/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732074/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b24d1486793d499db45cd198d63baec6

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi135 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732075]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:39:23
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732075/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732075/
sentry: https://sentry.ceph.com/organizations/ceph/?query=19e65144386d4bc6b79bc63628588403

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi169 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732076]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:50:16
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732076/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732076/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3789d2b95366452d9a367dc7677c989b

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi197 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732077]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:32:13
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732077/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732077/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5c2cdc0cb03f4cedb59112515e7cbda9

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi166 with
    status 2: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732078]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:29:05
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732078/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732078/
sentry: https://sentry.ceph.com/organizations/ceph/?query=350f04126adc479cbb5a79b69b0610ab

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi184 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6732079]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:27:25
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732079/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732079/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dc8ebac0e72f4e179ba8a725ea46a122

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi106 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732080]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:38:06
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732080/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732080/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d2d90da31a5f465184f3afc32f9f0757

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi081 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6732081]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:39:14
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732081/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732081/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a555a986ae648fca98902bdd7dd01ba

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi111 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732082]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:38:32
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732082/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732082/
sentry: https://sentry.ceph.com/organizations/ceph/?query=213ee010a530489a9ecb31e86469decb

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi119 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732083]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:38:27
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732083/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732083/
sentry: https://sentry.ceph.com/organizations/ceph/?query=024b2e3a59eb495195bbccd37170b5ef

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi183 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6732084]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:39:08
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732084/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732084/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1f692a06f679413c80cbc0ecabb685e9

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi181 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732085]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:22:28
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732085/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732085/
sentry: https://sentry.ceph.com/organizations/ceph/?query=822fa5a25a6f472bab5e28942f31312e

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi167 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732086]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:38:42
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732086/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732086/
sentry: https://sentry.ceph.com/organizations/ceph/?query=238a6871234f4838abd53fb9b6e40bc5

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi153 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6732087]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:17:53
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732087/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732087/
sentry: https://sentry.ceph.com/organizations/ceph/?query=128c7d60ed1644dbab43a671e4182276

    "2022-03-11T17:59:42.345881+0000 mon.a (mon.0) 94 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6732088]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:38:27
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732088/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732088/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d5a70fb4b4954ef7a678d9c7dd914439

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi196 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6732089]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:19:13
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732089/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732089/
sentry: https://sentry.ceph.com/organizations/ceph/?query=042a9806ba434ff595a26c7c8ae7c913

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi189 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6732090]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:20:52
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732090/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-11_15:56:57-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6732090/
sentry: https://sentry.ceph.com/organizations/ceph/?query=85521e8f29f64991b19516aecc09307a

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi152 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=f0baaf3694eacaeddea5f2758fc58361770a25c5
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'