2022-05-19T09:40:28.743 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-19T09:40:28.744 INFO:teuthology.results:seed: '9095' 2022-05-19T12:05:30.557 INFO:root:teuthology version: 1.1.1.dev333+gb018fa01 2022-05-19T12:05:30.562 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-19T12:41:33.505 INFO:teuthology.results:Tests finished! gathering results... 2022-05-19T12:41:34.935 INFO:scrape:Found 18 jobs 2022-05-19T12:41:37.263 INFO:scrape:Found 3 distinct failure reasons 2022-05-19T12:41:37.264 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-05-19T12:41:37.265 INFO:scrape:2 jobs: ['6838417', '6838422'] 2022-05-19T12:41:37.265 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-05-19T12:41:37.266 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-05-19T12:41:37.266 INFO:scrape: 2022-05-19T12:41:37.267 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-05-19T12:41:37.267 INFO:scrape:1 jobs: ['6838418'] 2022-05-19T12:41:37.267 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-05-19T12:41:37.268 INFO:scrape: 2022-05-19T12:41:37.269 INFO:scrape:Failure: "2022-05-19T10:05:22.708141+0000 mon.a (mon.0) 273 : cluster [WRN] Health check failed: Degraded data redundancy: 3/146 objects degraded (2.055%), 3 pgs degraded (PG_DEGRADED)" in cluster log 2022-05-19T12:41:37.269 INFO:scrape:2 jobs: ['6838416', '6838426'] 2022-05-19T12:41:37.270 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-05-19T12:41:37.270 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-05-19T12:41:37.271 INFO:scrape: 2022-05-19T12:41:37.273 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [6838416] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:29:47 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838416/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838416/ "2022-05-19T10:05:22.708141+0000 mon.a (mon.0) 273 : cluster [WRN] Health check failed: Degraded data redundancy: 3/146 objects degraded (2.055%), 3 pgs degraded (PG_DEGRADED)" in cluster log [6838417] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:38:24 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838417/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838417/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a45aca363024472f91f4435b0021b593 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6838418] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:18:55 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838418/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46040468504345a680657149abc0979b reached maximum tries (90) after waiting for 540 seconds [6838422] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:34:06 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838422/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838422/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c99a637eedc4518bac0244674ff064e Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi158 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f4e100b4f5410103dffd08bb1b8825954848fa4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6838426] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:20:09 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838426/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=25a2b535c08e4c049229e0fe5e919028 "2022-05-19T10:17:20.926129+0000 mon.a (mon.0) 90 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log Pass ================================================================= [6838412] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:31:07 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838412/ [6838413] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:52:02 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838413/ [6838414] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:30:41 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838414/ [6838415] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} time: 01:04:56 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838415/ [6838419] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}} time: 00:31:29 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838419/ [6838420] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} time: 00:51:47 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838420/ [6838421] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:28:32 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838421/ [6838423] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}} time: 00:52:25 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838423/ [6838424] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}} time: 00:48:44 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838424/ [6838425] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:59:08 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838425/ [6838427] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:52:00 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838427/ [6838428] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:29:58 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838428/ [6838429] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}} time: 00:30:48 info: http://pulpito.ceph.com/mrfreezeex-2022-05-17_13:42:04-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6838429/