2022-05-09T14:06:20.257 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01
2022-05-09T14:06:20.258 INFO:teuthology.results:seed: '8227'
2022-05-09T15:19:09.174 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01
2022-05-09T15:19:09.195 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-05-09T16:14:13.258 INFO:teuthology.results:Tests finished! gathering results...
2022-05-09T16:14:14.360 INFO:scrape:Found 16 jobs
2022-05-09T16:14:17.705 INFO:scrape:Found 2 distinct failure reasons
2022-05-09T16:14:17.706 INFO:scrape:Failure: "2022-05-09T15:23:33.965219+0000 mon.a (mon.0) 282 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log
2022-05-09T16:14:17.706 INFO:scrape:1 jobs: ['6828748']
2022-05-09T16:14:17.707 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-05-09T16:14:17.707 INFO:scrape:
2022-05-09T16:14:17.708 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi097 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c6bf619fb466bf4e45604b8a9970e29736aeb769 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-05-09T16:14:17.708 INFO:scrape:6 jobs: ['6828745', '6828746', '6828747', '6828753', '6828741', '6828754']
2022-05-09T16:14:17.709 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-05-09T16:14:17.709 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-05-09T16:14:17.710 INFO:scrape:
2022-05-09T16:14:17.711 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
failed:  7
dead:    0
running: 0
waiting: 0
queued:  0
passed:  9


Fail
=================================================================
[6828741]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:27:12
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828741/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828741/
sentry: https://sentry.ceph.com/organizations/ceph/?query=afcf6ec4f42049b2a8b64f8c46cfc5f8

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi145 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c6bf619fb466bf4e45604b8a9970e29736aeb769
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6828745]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:25:18
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828745/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828745/
sentry: https://sentry.ceph.com/organizations/ceph/?query=03df51c305444fa3804ce2eb71852d2e

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi097 with
    status 16: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c6bf619fb466bf4e45604b8a9970e29736aeb769
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6828746]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:25:39
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828746/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828746/
sentry: https://sentry.ceph.com/organizations/ceph/?query=54aeef6040ea4550886a8fb73206ad7a

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi092 with
    status 16: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c6bf619fb466bf4e45604b8a9970e29736aeb769
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6828747]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:26:30
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828747/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828747/
sentry: https://sentry.ceph.com/organizations/ceph/?query=301c5cb0f9534b1d9709935943fd00a2

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi124 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c6bf619fb466bf4e45604b8a9970e29736aeb769
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6828748]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:25:45
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828748/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828748/

    "2022-05-09T15:23:33.965219+0000 mon.a (mon.0) 282 : cluster [WRN] Health
    check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in
    cluster log

[6828753]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:30:22
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828753/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828753/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d10bff17981d40f8af69b562914ea5ec

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi188 with
    status 16: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c6bf619fb466bf4e45604b8a9970e29736aeb769
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

[6828754]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:27:03
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828754/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828754/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7254044d49de4de1bbacc408b982d32a

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi151 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c6bf619fb466bf4e45604b8a9970e29736aeb769
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'



Pass
=================================================================
[6828742] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}}
time:   00:50:42
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828742/

[6828743] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}}
time:   00:51:15
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828743/

[6828744] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8}}
time:   00:25:54
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828744/

[6828749] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}}
time:   00:48:20
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828749/

[6828750] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}}
time:   00:53:12
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828750/

[6828751] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}}
time:   00:24:46
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828751/

[6828752] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}}
time:   00:25:48
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828752/

[6828755] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}}
time:   00:25:31
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828755/

[6828756] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}}
time:   00:48:40
info:   http://pulpito.ceph.com/mrfreezeex-2022-05-09_13:22:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6828756/