2022-03-07T15:47:08.393 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-07T15:47:08.394 INFO:teuthology.results:seed: '1172'
2022-03-07T16:10:43.416 INFO:root:teuthology version: 1.1.0-eeaeee38
2022-03-07T16:10:43.449 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-07T16:31:45.021 INFO:teuthology.results:Tests finished! gathering results...
2022-03-07T16:31:46.104 INFO:scrape:Found 18 jobs
2022-03-07T16:31:49.331 INFO:scrape:Found 3 distinct failure reasons
2022-03-07T16:31:49.332 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-03-07T16:31:49.333 INFO:scrape:11 jobs: ['6725058', '6725055', '6725061', '6725071', '6725069', '6725067', '6725063', '6725070', '6725060', '6725056', '6725066']
2022-03-07T16:31:49.334 INFO:scrape:suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-07T16:31:49.334 INFO:scrape:suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-07T16:31:49.335 INFO:scrape:
2022-03-07T16:31:49.335 INFO:scrape:Failure: "2022-03-07T16:14:29.686985+0000 mon.a (mon.0) 94 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2022-03-07T16:31:49.336 INFO:scrape:1 jobs: ['6725064']
2022-03-07T16:31:49.336 INFO:scrape:suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-07T16:31:49.337 INFO:scrape:
2022-03-07T16:31:49.337 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
2022-03-07T16:31:49.338 INFO:scrape:6 jobs: ['6725065', '6725057', '6725059', '6725072', '6725062', '6725068']
2022-03-07T16:31:49.338 INFO:scrape:suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-07T16:31:49.339 INFO:scrape:suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-07T16:31:49.339 INFO:scrape:
2022-03-07T16:31:49.341 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
failed:  18
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6725055]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:13:04
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725055/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725055/
sentry: https://sentry.ceph.com/organizations/ceph/?query=174c48372a4349e6a7b4169d2d0f3473

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi083 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725056]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:11:23
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725056/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725056/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c32bf3e094a8446c8a58d44fa8645347

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi029 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725057]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}
-----------------------------------------------------------------
time:   00:09:41
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725057/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725057/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f978842aa6a41acaadb8321c14eebee

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi039 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6725058]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/none rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:12:26
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725058/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725058/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2bf7e001292f4092bbe0ae299cbd9269

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi099 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725059]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}
-----------------------------------------------------------------
time:   00:15:54
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725059/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725059/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8ef3f83e29a643a1a785ee8475a5cb0d

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi090 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6725060]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:13:42
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725060/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725060/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b71e500e587740038029a0a751052c67

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi073 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725061]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:10:27
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725061/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725061/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3182da6e320342b7b6fd7b6084534f9a

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi176 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725062]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}
-----------------------------------------------------------------
time:   00:10:30
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725062/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725062/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c5e8317b1c6d4d8b9b414a6e2f48650c

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi079 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6725063]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:13:19
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725063/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725063/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4b6370caad044baf95b77027c38a5e7a

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi049 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725064]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:15:37
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725064/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725064/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bd8b729bcc3e493e9a5f9e5872d79f18

    "2022-03-07T16:14:29.686985+0000 mon.a (mon.0) 94 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6725065]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}
-----------------------------------------------------------------
time:   00:16:13
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725065/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725065/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fd45e197c5b84847b63ccdbc2b410a6c

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi062 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6725066]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:13:25
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725066/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725066/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eba50b26097742418e6c26c099ecef18

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi061 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725067]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-hybrid policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:10:00
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725067/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725067/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c2ac25cd1919454ba1da5815251a8418

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi155 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725068]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}
-----------------------------------------------------------------
time:   00:16:11
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725068/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725068/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9adbaf3c7eea4c32b146193af4fdf1c4

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi112 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6725069]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:13:11
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725069/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725069/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0a3b111ac28443208d4fa7f38c8d0944

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi037 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725070]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:10:51
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725070/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725070/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a0219ea41cec43d0ab3da298ff633963

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi092 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725071]  rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/filestore-xfs policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-journal-workunit-multipeer}
-----------------------------------------------------------------
time:   00:12:23
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725071/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725071/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f4539aa043724f448c8ec0ba65af8f44

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi072 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6725072]  rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}
-----------------------------------------------------------------
time:   00:11:14
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725072/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-07_15:46:53-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6725072/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fc3165d59b224cb68a0ceca42a7e09d0

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi039 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=edb3cab60d387bc3f79bad1a9dc2a622ca074498
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'