2022-04-08T20:16:47.418 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-08T20:16:47.419 INFO:teuthology.results:seed: '6500' 2022-04-08T20:39:00.061 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-08T20:39:00.065 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-08T21:35:04.049 INFO:teuthology.results:Tests finished! gathering results... 2022-04-08T21:35:05.510 INFO:scrape:Found 20 jobs 2022-04-08T21:35:09.236 INFO:scrape:Found 2 distinct failure reasons 2022-04-08T21:35:09.237 INFO:scrape:Failure: "2022-04-08T20:34:52.787793+0000 mon.a (mon.0) 87 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-04-08T21:35:09.238 INFO:scrape:2 jobs: ['6782860', '6782865'] 2022-04-08T21:35:09.238 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-04-08T21:35:09.239 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-04-08T21:35:09.239 INFO:scrape: 2022-04-08T21:35:09.240 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi120 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-04-08T21:35:09.240 INFO:scrape:4 jobs: ['6782868', '6782873', '6782869', '6782861'] 2022-04-08T21:35:09.241 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-08T21:35:09.241 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-08T21:35:09.241 INFO:scrape: 2022-04-08T21:35:09.243 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 14 Fail ================================================================= [6782860] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:19:28 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782860/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782860/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eaf65310d36b425ab87ca44b17252da3 "2022-04-08T20:34:52.787793+0000 mon.a (mon.0) 87 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6782861] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 01:05:31 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782861/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=11d81c3c3d944255a90ec69b23035a48 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6782865] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:15:21 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782865/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e0c54d0c5ed451bb7a648d139da3640 "2022-04-08T20:38:36.176922+0000 mon.a (mon.0) 99 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6782868] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:28:47 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782868/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f3b882fc9c5481d9aa9b5f4bbfcfdde Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi120 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6782869] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:30:07 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782869/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782869/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b396caa2bb844d0b87683a699987cb73 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi142 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6782873] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:25:26 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782873/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8aee40e2150049afbb8ff8aef4fd5389 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi169 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' Pass ================================================================= [6782858] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:51:06 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782858/ [6782859] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:26:41 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782859/ [6782862] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8}} time: 00:26:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782862/ [6782863] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} time: 00:49:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782863/ [6782864] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}} time: 00:53:53 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782864/ [6782866] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} time: 00:53:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782866/ [6782867] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:25:01 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782867/ [6782870] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} time: 00:24:49 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782870/ [6782871] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}} time: 00:49:25 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782871/ [6782872] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}} time: 00:25:14 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782872/ [6782874] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}} time: 00:25:43 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782874/ [6782875] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{centos_8}} time: 00:45:02 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782875/ [6782876] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} time: 00:50:25 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782876/ [6782877] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:26:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-08_14:23:55-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6782877/