2022-03-15T15:19:32.870 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-15T15:19:32.872 INFO:teuthology.results:seed: '4419' 2022-03-15T15:28:30.905 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-15T15:28:30.910 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-15T16:33:35.588 INFO:teuthology.results:Tests finished! gathering results... 2022-03-15T16:33:36.844 INFO:scrape:Found 18 jobs 2022-03-15T16:33:46.945 INFO:scrape:Found 3 distinct failure reasons 2022-03-15T16:33:46.946 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-03-15T16:33:46.946 INFO:scrape:9 jobs: ['6738469', '6738470', '6738457', '6738461', '6738467', '6738456', '6738459', '6738464', '6738455'] 2022-03-15T16:33:46.947 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T16:33:46.947 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T16:33:46.948 INFO:scrape: 2022-03-15T16:33:46.948 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-03-15T16:33:46.949 INFO:scrape:6 jobs: ['6738462', '6738466', '6738471', '6738458', '6738468', '6738460'] 2022-03-15T16:33:46.949 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-15T16:33:46.950 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-15T16:33:46.950 INFO:scrape: 2022-03-15T16:33:46.951 INFO:scrape:Failure: "2022-03-15T15:38:51.663069+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-03-15T16:33:46.951 INFO:scrape:2 jobs: ['6738454', '6738465'] 2022-03-15T16:33:46.951 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'mirror', 'msgr-failures/few', 'roles}'] 2022-03-15T16:33:46.952 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-15T16:33:46.952 INFO:scrape: 2022-03-15T16:33:46.954 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 17 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6738454] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:18:28 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738454/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738454/ sentry: https://sentry.ceph.com/organizations/ceph/?query=014407cb88774d349abcb8695c6bc8fb "2022-03-15T15:38:51.663069+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6738455] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:58:30 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738455/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738455/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64d3763c95754768808abae0fc4b66f5 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738456] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:42:45 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738456/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738456/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1cd753e2b6d43658f2493aeb5a1869d Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi195 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738457] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:50:44 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738457/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00372da2aa744d768c30317148d7450f Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi146 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738458] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:43:00 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738458/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738458/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9438c239c0b44198dc91653a98beae2 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738459] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:49:05 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738459/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738459/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f148669536a34205b28b8b85c5bb08b3 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738460] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:42:45 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738460/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75776197b7664fa7916cab8ed6a047bd Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738461] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:26:21 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738461/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738461/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc1ce630ce8c4957b287cd8bb3794cb7 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi187 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738462] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:41:46 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738462/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e086c35b1e084066923eec8346662917 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738464] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:44:08 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738464/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=714f4bcfa402467f8a70a56338fadffb Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi109 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738465] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:17:23 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738465/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8eb98473e89e4e30afd271a671a4e4fe "2022-03-15T15:41:31.569469+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6738466] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:43:10 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738466/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83e7ac82c7604f7280f4573e2336c0be Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738467] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:43:39 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738467/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738467/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7b14a96493f41f580a71366a9714323 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi197 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738468] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:42:22 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738468/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738468/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e9e46ede0e744178ddcfa030478150f Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi184 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738469] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:42:56 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738469/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78f4fc21b453437f9235db548e86f118 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738470] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:55:44 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738470/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b4ce447f31ce41ea9c08d61a59b6ef68 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738471] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:29:16 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738471/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5fa9f76ae1c240e78fb62621e8e0c01c Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e479fde70ca281b264661a561f10d7b490dc1dfb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' Pass ================================================================= [6738463] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} time: 00:28:11 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_15:18:15-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738463/