2022-06-08T18:19:08.733 INFO:root:teuthology version: 1.1.1.dev364+g40fff4b2
2022-06-08T18:19:08.734 INFO:teuthology.results:seed: '7745'
2022-06-08T18:42:51.812 INFO:root:teuthology version: 1.1.1.dev364+g40fff4b2
2022-06-08T18:42:51.836 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-06-08T19:42:56.647 INFO:teuthology.results:Tests finished! gathering results...
2022-06-08T19:42:58.094 INFO:scrape:Found 21 jobs
2022-06-08T19:42:58.845 WARNING:scrape:Not-understood crash indication 2022-06-08T19:13:46.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi093.stderr:daemon-helper: command crashed with signal 11
2022-06-08T19:42:58.846 WARNING:scrape:Not-understood crash indication 
2022-06-08T19:42:59.201 WARNING:scrape:Not-understood crash indication 2022-06-08T19:13:46.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi093.stderr:daemon-helper: command crashed with signal 11
2022-06-08T19:42:59.202 WARNING:scrape:Not-understood crash indication 
2022-06-08T19:42:59.565 WARNING:scrape:Not-understood crash indication 2022-06-08T19:13:46.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi093.stderr:daemon-helper: command crashed with signal 11
2022-06-08T19:42:59.567 WARNING:scrape:Not-understood crash indication 
2022-06-08T19:43:00.678 INFO:scrape:Found 4 distinct failure reasons
2022-06-08T19:43:00.679 INFO:scrape:Failure: Found coredumps on ubuntu@smithi093.front.sepia.ceph.com
2022-06-08T19:43:00.679 INFO:scrape:1 jobs: ['6868473']
2022-06-08T19:43:00.679 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-06-08T19:43:00.680 INFO:scrape:
2022-06-08T19:43:00.680 INFO:scrape:Failure: "2022-06-08T18:35:46.787648+0000 mon.a (mon.0) 98 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log
2022-06-08T19:43:00.681 INFO:scrape:1 jobs: ['6868478']
2022-06-08T19:43:00.681 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}']
2022-06-08T19:43:00.682 INFO:scrape:
2022-06-08T19:43:00.682 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-06-08T19:43:00.682 INFO:scrape:1 jobs: ['6868476']
2022-06-08T19:43:00.683 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-06-08T19:43:00.683 INFO:scrape:
2022-06-08T19:43:00.684 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi185 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2302942357d40471993bc52109ae57214f768c40 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'
2022-06-08T19:43:00.684 INFO:scrape:1 jobs: ['6868479']
2022-06-08T19:43:00.685 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']
2022-06-08T19:43:00.685 INFO:scrape:
2022-06-08T19:43:00.687 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/
failed:  4
dead:    0
running: 0
waiting: 0
queued:  0
passed:  17


Fail
=================================================================
[6868473]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}}
-----------------------------------------------------------------
time:   00:59:26
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868473/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868473/

    Found coredumps on ubuntu@smithi093.front.sepia.ceph.com

[6868476]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:17:20
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868476/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868476/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6a1c18eb599d47c8b0296cacedf39730

    reached maximum tries (90) after waiting for 540 seconds

[6868478]  rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:16:12
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868478/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868478/
sentry: https://sentry.ceph.com/organizations/ceph/?query=bb1e2a05dbb249b3ae6442ae43a97aac

    "2022-06-08T18:35:46.787648+0000 mon.a (mon.0) 98 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6868479]  rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{ubuntu_latest}}
-----------------------------------------------------------------
time:   00:26:51
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868479/
log:    http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868479/
sentry: https://sentry.ceph.com/organizations/ceph/?query=954810ebf1a14cd3a15420635c1fec29

    Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi185 with
    status 1: 'mkdir -p --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd --
    /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=2302942357d40471993bc52109ae57214f768c40
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1"
    CEPH_ID="mirror" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror
    CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\'
    RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1
    RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'



Pass
=================================================================
[6868474] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}}
time:   00:27:17
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868474/

[6868475] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}}
time:   00:50:03
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868475/

[6868477] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}}
time:   00:46:19
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868477/

[6868480] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}}
time:   00:57:27
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868480/

[6868481] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}}
time:   00:29:09
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868481/

[6868482] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}}
time:   00:53:37
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868482/

[6868483] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}}
time:   00:28:13
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868483/

[6868484] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}}
time:   00:56:53
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868484/

[6868485] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}}
time:   00:52:31
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868485/

[6868486] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}}
time:   00:51:29
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868486/

[6868487] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}}
time:   00:27:24
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868487/

[6868488] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}}
time:   00:48:34
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868488/

[6868490] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}}
time:   00:56:05
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868490/

[6868492] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}}
time:   00:27:37
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868492/

[6868494] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{ubuntu_latest}}
time:   00:55:21
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868494/

[6868495] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}}
time:   00:26:37
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868495/

[6868497] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}}
time:   00:53:05
info:   http://pulpito.ceph.com/mrfreezeex-2022-06-08_15:09:21-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6868497/