2022-04-10T01:41:58.052 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-10T01:41:58.053 INFO:teuthology.results:seed: '7062' 2022-04-10T04:32:01.839 INFO:root:teuthology version: 1.1.0-a347b2ac 2022-04-10T04:32:01.847 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-10T09:58:26.469 INFO:teuthology.results:Tests finished! gathering results... 2022-04-10T09:58:42.158 INFO:scrape:Found 161 jobs 2022-04-10T09:58:46.398 INFO:scrape:Found 5 distinct failure reasons 2022-04-10T09:58:46.400 INFO:scrape:Failure: "2022-04-10T03:44:48.210560+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-04-10T09:58:46.400 INFO:scrape:8 jobs: ['6784596', '6784496', '6784532', '6784611', '6784645', '6784530', '6784512', '6784534'] 2022-04-10T09:58:46.401 INFO:scrape:suites intersection: ['cluster/2-nodes/{base/{install', 'mirror', 'msgr-failures/few', 'roles}'] 2022-04-10T09:58:46.401 INFO:scrape:suites union: ['cluster/2-nodes/{base/{install', 'extra/{mirror-extra}', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-journal-bootstrap-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'workloads/rbd-mirror-workunit-config-key}'] 2022-04-10T09:58:46.402 INFO:scrape: 2022-04-10T09:58:46.402 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi184 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-04-10T09:58:46.403 INFO:scrape:1 jobs: ['6784646'] 2022-04-10T09:58:46.403 INFO:scrape:suites: ['cluster/2-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-minimum}'] 2022-04-10T09:58:46.404 INFO:scrape: 2022-04-10T09:58:46.404 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-04-10T09:58:46.404 INFO:scrape:1 jobs: ['6784513'] 2022-04-10T09:58:46.405 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-10T09:58:46.405 INFO:scrape: 2022-04-10T09:58:46.406 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi153 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-04-10T09:58:46.406 INFO:scrape:3 jobs: ['6784545', '6784538', '6784584'] 2022-04-10T09:58:46.407 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-10T09:58:46.407 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-10T09:58:46.408 INFO:scrape: 2022-04-10T09:58:46.408 INFO:scrape:Dead: b'2022-04-10T09:50:16.597 DEBUG:teuthology.exit:Finished running handlers' 2022-04-10T09:58:46.409 INFO:scrape:ceph version 17.0.0-11492-g9b3739e7 (9b3739e792ef182b79445b926c7b7da461c05f52) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7faf15d543c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x555b2fc1718b] 5: (KernelDevice::_aio_thread()+0xe19) [0x555b308f7c49] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x555b30902971] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7faf15d48609] 8: clone() 2022-04-10T09:58:46.409 INFO:scrape:1 jobs: ['6784582'] 2022-04-10T09:58:46.410 INFO:scrape:suites: ['cluster/2-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] 2022-04-10T09:58:46.410 INFO:scrape: 2022-04-10T09:58:46.417 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 13 dead: 1 running: 0 waiting: 0 queued: 0 passed: 147 Fail ================================================================= [6784496] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-bootstrap-workunit} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:18:21 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784496/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784496/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04b026fe180e4775b52582b50226fa25 "2022-04-10T02:02:00.707334+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6784512] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:41:54 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784512/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784512/ "2022-04-10T02:17:33.451766+0000 mon.a (mon.0) 213 : cluster [WRN] Health check failed: Degraded data redundancy: 1/3696 objects degraded (0.027%), 1 pg degraded (PG_DEGRADED)" in cluster log [6784513] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:19:20 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784513/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784513/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3061e2a544754b939a73a72badfefe39 reached maximum tries (90) after waiting for 540 seconds [6784530] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:15:08 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784530/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784530/ sentry: https://sentry.ceph.com/organizations/ceph/?query=38bb6b985b804b62b53b55d3e30b8806 "2022-04-10T02:28:52.793618+0000 mon.a (mon.0) 89 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6784532] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-bootstrap-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:08:25 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784532/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784532/ "2022-04-10T02:34:24.210477+0000 mon.a (mon.0) 192 : cluster [WRN] Health check failed: Degraded data redundancy: 3/132 objects degraded (2.273%), 3 pgs degraded (PG_DEGRADED)" in cluster log [6784534] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:46:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784534/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784534/ "2022-04-10T02:41:35.852836+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [6784538] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:25:17 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784538/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784538/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b5053fc6445f488e8b33dee32844060a Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi174 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6784545] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:25:23 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784545/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784545/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d756e5a60226450f8d8f011fda331b7b Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi153 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6784584] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:27:28 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784584/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784584/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d3e5cd9d3334902af644335e812946f Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi173 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6784596] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-config-key} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:18:16 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784596/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784596/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd3dfa7061894d11a066dd1da1f481d7 "2022-04-10T03:44:48.210560+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6784611] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:40:53 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784611/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784611/ "2022-04-10T04:04:12.245290+0000 mon.a (mon.0) 227 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [6784645] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:40:26 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784645/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784645/ "2022-04-10T04:40:19.845783+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Degraded data redundancy: 1/2670 objects degraded (0.037%), 1 pg degraded (PG_DEGRADED)" in cluster log [6784646] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:17:26 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784646/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784646/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b04ee415ab5142089d5aa0575524bcdb Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi184 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3739e792ef182b79445b926c7b7da461c05f52 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' Dead ================================================================= [6784582] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784582/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784582/ hit max job timeout Pass ================================================================= [6784493] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-ha-workunit} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:14:10 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784493/ [6784494] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:56:51 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784494/ [6784495] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} time: 00:55:01 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784495/ [6784497] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} time: 00:50:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784497/ [6784498] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:21:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784498/ [6784499] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-journaling} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:19:42 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784499/ [6784500] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:39:52 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784500/ [6784501] rbd/singleton/{all/rbd_mirror objectstore/bluestore-comp-snappy openstack supported-random-distro$/{centos_8}} time: 00:44:40 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784501/ [6784502] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:47:38 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784502/ [6784503] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:56:40 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784503/ [6784504] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} time: 00:37:18 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784504/ [6784505] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}} time: 00:27:22 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784505/ [6784506] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} time: 00:49:29 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784506/ [6784507] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-policy-none} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:28:02 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784507/ [6784508] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}} time: 00:47:33 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784508/ [6784509] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-stupid validator/memcheck workloads/rbd_mirror} time: 00:54:39 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784509/ [6784510] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-policy-simple} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} time: 00:28:48 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784510/ [6784511] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} time: 00:47:52 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784511/ [6784514] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:26:53 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784514/ [6784515] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} time: 00:56:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784515/ [6784516] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:39:18 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784516/ [6784517] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} time: 00:45:56 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784517/ [6784518] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:19:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784518/ [6784519] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:48:34 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784519/ [6784520] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:20:22 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784520/ [6784521] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} time: 00:47:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784521/ [6784522] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:19:48 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784522/ [6784523] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}} time: 00:49:23 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784523/ [6784524] rbd/singleton/{all/rbd_mirror objectstore/bluestore-comp-zlib openstack supported-random-distro$/{centos_8}} time: 00:44:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784524/ [6784525] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} time: 00:53:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784525/ [6784526] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} time: 00:57:21 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784526/ [6784527] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} time: 00:27:41 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784527/ [6784528] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-low-osd-mem-target validator/memcheck workloads/rbd_mirror} time: 00:54:13 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784528/ [6784529] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-policy-simple} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}} time: 00:28:30 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784529/ [6784531] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} time: 00:56:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784531/ [6784533] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:08:59 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784533/ [6784535] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} time: 00:19:53 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784535/ [6784536] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{centos_8}} time: 00:47:12 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784536/ [6784537] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-journaling} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:21:12 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784537/ [6784539] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}} time: 00:57:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784539/ [6784540] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-config-key} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}} time: 00:29:16 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784540/ [6784541] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:27:51 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784541/ [6784542] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{centos_8}} time: 00:47:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784542/ [6784543] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} time: 00:47:35 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784543/ [6784544] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-hybrid validator/memcheck workloads/rbd_mirror} time: 00:54:43 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784544/ [6784546] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}} time: 00:26:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784546/ [6784547] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}} time: 00:56:42 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784547/ [6784548] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-bootstrap-workunit} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} time: 00:08:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784548/ [6784549] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} time: 00:41:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784549/ [6784550] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}} time: 00:09:56 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784550/ [6784551] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} time: 00:46:12 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784551/ [6784552] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} time: 00:21:13 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784552/ [6784553] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:47:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784553/ [6784554] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}} time: 00:21:49 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784554/ [6784555] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} time: 00:46:09 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784555/ [6784556] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-journaling} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8}} time: 00:21:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784556/ [6784557] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:38:59 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784557/ [6784558] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:53:04 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784558/ [6784559] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{centos_8}} time: 00:37:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784559/ [6784560] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest}} time: 00:27:28 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784560/ [6784561] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}} time: 00:49:41 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784561/ [6784562] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} time: 00:46:31 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784562/ [6784563] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zstd validator/memcheck workloads/rbd_mirror} time: 00:55:20 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784563/ [6784564] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:46:04 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784564/ [6784565] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-ha-workunit} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}} time: 00:14:09 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784565/ [6784566] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} time: 00:37:39 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784566/ [6784567] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:25:16 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784567/ [6784568] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}} time: 00:56:38 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784568/ [6784569] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-bootstrap-workunit} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:08:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784569/ [6784570] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:38:42 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784570/ [6784571] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}} time: 00:08:16 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784571/ [6784572] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-minimum} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:20:30 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784572/ [6784573] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}} time: 00:50:03 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784573/ [6784574] rbd/singleton/{all/rbd_mirror objectstore/bluestore-low-osd-mem-target openstack supported-random-distro$/{centos_8}} time: 00:44:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784574/ [6784575] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}} time: 00:51:55 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784575/ [6784576] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} time: 00:57:38 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784576/ [6784577] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} time: 00:36:10 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784577/ [6784578] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} time: 00:48:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784578/ [6784579] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-policy-none} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:28:35 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784579/ [6784580] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{ubuntu_latest}} time: 00:46:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784580/ [6784581] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-zlib validator/memcheck workloads/rbd_mirror} time: 00:54:33 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784581/ [6784583] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} time: 00:38:35 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784583/ [6784585] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} time: 00:26:04 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784585/ [6784586] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{ubuntu_latest}} time: 00:54:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784586/ [6784587] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:37:05 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784587/ [6784588] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:10:11 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784588/ [6784589] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{ubuntu_latest}} time: 00:46:56 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784589/ [6784590] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} time: 00:47:48 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784590/ [6784591] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-journaling} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}} time: 00:20:38 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784591/ [6784592] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:40:01 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784592/ [6784593] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-minimum} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:19:49 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784593/ [6784594] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}} time: 00:55:33 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784594/ [6784595] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} time: 01:01:13 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784595/ [6784597] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}} time: 00:34:51 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784597/ [6784598] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} time: 00:47:57 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784598/ [6784599] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-policy-none} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}} time: 00:28:59 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784599/ [6784600] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-snappy validator/memcheck workloads/rbd_mirror} time: 00:54:53 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784600/ [6784601] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} time: 00:48:11 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784601/ [6784602] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-ha-workunit} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:13:59 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784602/ [6784603] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} time: 00:45:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784603/ [6784604] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-bootstrap-workunit} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}} time: 00:10:12 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784604/ [6784605] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:39:37 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784605/ [6784606] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:08:26 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784606/ [6784607] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{centos_8}} time: 00:45:42 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784607/ [6784608] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:46:48 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784608/ [6784609] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}} time: 00:21:18 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784609/ [6784610] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-journaling} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} time: 00:21:25 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784610/ [6784612] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest}} time: 00:19:39 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784612/ [6784613] rbd/singleton/{all/rbd_mirror objectstore/filestore-xfs openstack supported-random-distro$/{centos_8}} time: 00:44:30 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784613/ [6784614] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} time: 00:36:29 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784614/ [6784615] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}} time: 00:28:19 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784615/ [6784616] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-lz4 validator/memcheck workloads/rbd_mirror} time: 00:54:26 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784616/ [6784617] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-policy-simple} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:27:35 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784617/ [6784618] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:47:02 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784618/ [6784619] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} time: 00:50:52 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784619/ [6784620] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:25:29 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784620/ [6784621] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} time: 00:54:02 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784621/ [6784622] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:38:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784622/ [6784623] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} time: 00:46:52 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784623/ [6784624] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} time: 00:19:41 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784624/ [6784625] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}} time: 00:21:19 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784625/ [6784626] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-minimum} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:19:25 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784626/ [6784627] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-stress-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:51:11 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784627/ [6784628] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:54:20 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784628/ [6784629] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-config-key} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:28:41 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784629/ [6784630] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}} time: 00:29:08 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784630/ [6784631] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-policy-none} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} time: 00:28:34 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784631/ [6784632] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} time: 00:48:03 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784632/ [6784633] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-bitmap validator/memcheck workloads/rbd_mirror} time: 00:54:29 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784633/ [6784634] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:48:01 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784634/ [6784635] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-ha-workunit} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} time: 00:13:48 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784635/ [6784636] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} time: 00:43:11 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784636/ [6784637] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:51:17 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784637/ [6784638] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:24:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784638/ [6784639] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-stress-workunit} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}} time: 00:55:34 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784639/ [6784640] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:36:56 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784640/ [6784641] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}} time: 00:10:23 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784641/ [6784642] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}} time: 00:46:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784642/ [6784643] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} time: 00:48:49 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784643/ [6784644] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:21:53 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784644/ [6784647] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}} time: 00:50:54 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784647/ [6784648] rbd/singleton/{all/rbd_mirror objectstore/bluestore-comp-lz4 openstack supported-random-distro$/{ubuntu_latest}} time: 00:43:36 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784648/ [6784649] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} time: 00:38:05 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784649/ [6784650] rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-workunit-min-compat-client-octopus} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:27:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784650/ [6784651] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:50:29 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784651/ [6784652] rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/filestore-xfs validator/memcheck workloads/rbd_mirror} time: 00:53:59 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784652/ [6784653] rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:51:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-09_15:57:25-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6784653/