2022-05-17T16:22:23.855 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-17T16:22:23.856 INFO:teuthology.results:seed: '3042' 2022-05-17T16:50:26.148 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-17T16:50:26.160 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-17T18:01:31.606 INFO:teuthology.results:Tests finished! gathering results... 2022-05-17T18:01:33.270 INFO:scrape:Found 18 jobs 2022-05-17T18:01:34.996 INFO:scrape:Found 2 distinct failure reasons 2022-05-17T18:01:34.997 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-05-17T18:01:34.998 INFO:scrape:4 jobs: ['6838398', '6838404', '6838407', '6838401'] 2022-05-17T18:01:34.998 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-05-17T18:01:34.998 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-05-17T18:01:34.999 INFO:scrape: 2022-05-17T18:01:34.999 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2022-05-17T18:01:35.000 INFO:scrape:1 jobs: ['6838406'] 2022-05-17T18:01:35.000 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-05-17T18:01:35.001 INFO:scrape: 2022-05-17T18:01:35.002 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [6838398] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:31:06 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838398/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7e7662656d8c4c1893d7c111e9fb7172 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi188 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6838401] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:31:27 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838401/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=30e3ad6a5f824913b4b5801f2eb5add3 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi189 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6838404] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:35:25 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838404/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838404/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e8c28934c5f4ee4bfca0050233ddbf1 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi079 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6838406] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:59:09 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838406/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838406/ reached maximum tries (50) after waiting for 300 seconds [6838407] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 01:01:55 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838407/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eab8e6d2e3a341848a3e5b72cc4acc01 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' Pass ================================================================= [6838392] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:30:49 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838392/ [6838393] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}} time: 00:51:17 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838393/ [6838394] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}} time: 00:58:22 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838394/ [6838395] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:30:55 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838395/ [6838396] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} time: 00:51:51 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838396/ [6838397] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:28:47 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838397/ [6838399] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}} time: 00:52:53 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838399/ [6838400] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}} time: 00:30:48 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838400/ [6838402] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}} time: 00:30:45 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838402/ [6838403] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} time: 00:52:10 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838403/ [6838405] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} time: 00:31:02 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838405/ [6838408] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:28:12 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838408/ [6838409] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} time: 00:58:47 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:40:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838409/