2022-03-15T16:23:32.610 INFO:root:teuthology version: 1.1.0-fbfc7b38
2022-03-15T16:23:32.612 INFO:teuthology.results:seed: '9822'
2022-03-15T16:37:06.600 INFO:root:teuthology version: 1.1.0-fbfc7b38
2022-03-15T16:37:06.627 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-03-15T17:27:10.208 INFO:teuthology.results:Tests finished! gathering results...
2022-03-15T17:27:11.129 INFO:scrape:Found 16 jobs
2022-03-15T17:27:16.959 INFO:scrape:Found 3 distinct failure reasons
2022-03-15T17:27:16.960 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-03-15T17:27:16.961 INFO:scrape:9 jobs: ['6738490', '6738487', '6738482', '6738494', '6738492', '6738488', '6738485', '6738483', '6738489']
2022-03-15T17:27:16.961 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-15T17:27:16.962 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-03-15T17:27:16.962 INFO:scrape:
2022-03-15T17:27:16.963 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'
2022-03-15T17:27:16.963 INFO:scrape:6 jobs: ['6738493', '6738480', '6738484', '6738491', '6738481', '6738495']
2022-03-15T17:27:16.964 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-15T17:27:16.964 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-15T17:27:16.965 INFO:scrape:
2022-03-15T17:27:16.965 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-03-15T17:27:16.966 INFO:scrape:1 jobs: ['6738486']
2022-03-15T17:27:16.966 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-03-15T17:27:16.967 INFO:scrape:
2022-03-15T17:27:16.968 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
failed:  16
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[6738480]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:41:58
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738480/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738480/
sentry: https://sentry.ceph.com/organizations/ceph/?query=976bf74ba72a4167a3f186a1d25f6b43

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi139 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6738481]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:42:00
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738481/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738481/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f61053ba8494d7eb35e60dbf687ff41

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi187 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6738482]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:43:04
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738482/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738482/
sentry: https://sentry.ceph.com/organizations/ceph/?query=f247cea42aa247cd9c0f95d2f3103c33

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi201 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738483]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:41:25
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738483/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738483/
sentry: https://sentry.ceph.com/organizations/ceph/?query=22dfa8400952457c8acfc6d6520cd1b3

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi085 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738484]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:42:04
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738484/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738484/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a290bb03b112426c87e2168d1fad6d95

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi121 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6738485]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:43:10
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738485/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738485/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9545bdc295b5471a87530e42cee44eb2

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi155 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738486]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:19:10
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738486/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738486/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6db48b5afd2b4f8ea0807cefd440b670

    reached maximum tries (90) after waiting for 540 seconds

[6738487]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:42:00
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738487/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738487/
sentry: https://sentry.ceph.com/organizations/ceph/?query=71b3dc78f4244284947de80d3300308d

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi141 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738488]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:42:23
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738488/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738488/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8b902b70ad0e4188b85337d4c3af84da

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi142 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738489]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:44:33
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738489/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738489/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0568238d07cf4595bd9cacc37106886c

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi186 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738490]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:43:15
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738490/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738490/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a35a38bdb56e48888adb0d62e9213ec6

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi176 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738491]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:32:52
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738491/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738491/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fffd589fb24c4a7bad6a134591b8795b

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi173 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6738492]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:42:54
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738492/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738492/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4815b34c36624592bdff6eb594e26e3b

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi171 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738493]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:42:26
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738493/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738493/
sentry: https://sentry.ceph.com/organizations/ceph/?query=04b4897161834cac9f7711fa9a517576

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi123 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'

[6738494]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:42:03
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738494/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738494/
sentry: https://sentry.ceph.com/organizations/ceph/?query=da5ba1308c02498b9a4bceb6f50207e3

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi204 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6738495]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:32:37
info:   http://pulpito.ceph.com/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738495/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_16:22:42-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738495/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e08f87ebefe34499b641d8085d3d9d4f

    Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi159 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=4b642841e6422bf5c13e04401bc1dfe6b7fcfba8
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff
    RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4
    RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep
    htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh'