2022-04-09T21:21:41.097 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-09T21:21:41.099 INFO:teuthology.results:seed: '4439' 2022-04-09T22:02:34.249 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-09T22:02:34.263 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-09T22:59:38.365 INFO:teuthology.results:Tests finished! gathering results... 2022-04-09T22:59:39.419 INFO:scrape:Found 17 jobs 2022-04-09T22:59:41.988 INFO:scrape:Found 2 distinct failure reasons 2022-04-09T22:59:41.989 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi145 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-04-09T22:59:41.990 INFO:scrape:1 jobs: ['6784290'] 2022-04-09T22:59:41.991 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-09T22:59:41.992 INFO:scrape: 2022-04-09T22:59:41.993 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2022-04-09T22:59:41.994 INFO:scrape:2 jobs: ['6784280', '6784292'] 2022-04-09T22:59:41.995 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-09T22:59:41.996 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-09T22:59:41.996 INFO:scrape: 2022-04-09T22:59:41.999 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [6784280] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:58:49 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784280/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784280/ reached maximum tries (50) after waiting for 300 seconds [6784290] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:22:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784290/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784290/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9aeae6bd32a40fca08bf8caea201d3f Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi145 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6784292] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:15:30 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784292/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784292/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7050de5b6fad4570a5cdfe6dbb8792df reached maximum tries (90) after waiting for 540 seconds Pass ================================================================= [6784278] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}} time: 00:58:56 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784278/ [6784279] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:27:07 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784279/ [6784281] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}} time: 00:48:31 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784281/ [6784282] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} time: 00:52:28 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784282/ [6784283] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} time: 00:50:20 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784283/ [6784284] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}} time: 00:53:01 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784284/ [6784285] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} time: 00:46:48 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784285/ [6784286] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:50:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784286/ [6784287] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} time: 00:26:08 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784287/ [6784288] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}} time: 00:57:12 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784288/ [6784289] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} time: 00:52:46 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784289/ [6784291] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} time: 00:53:08 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784291/ [6784293] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:25:27 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784293/ [6784294] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}} time: 00:50:54 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:52:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784294/