2022-04-06T12:45:23.353 INFO:root:teuthology version: 1.1.0-3dd79763 2022-04-06T12:45:23.354 INFO:teuthology.results:seed: '7748' 2022-04-07T04:12:15.255 INFO:root:teuthology version: 1.1.0-3dd79763 2022-04-07T04:12:15.269 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-07T10:51:43.356 INFO:teuthology.results:Tests finished! gathering results... 2022-04-07T10:51:45.368 INFO:scrape:Found 18 jobs 2022-04-07T10:51:54.188 INFO:scrape:Found 6 distinct failure reasons 2022-04-07T10:51:54.189 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-04-07T10:51:54.189 INFO:scrape:2 jobs: ['6779090', '6779095'] 2022-04-07T10:51:54.190 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.190 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.191 INFO:scrape: 2022-04-07T10:51:54.191 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-04-07T10:51:54.192 INFO:scrape:5 jobs: ['6779088', '6779091', '6779081', '6779084', '6779085'] 2022-04-07T10:51:54.192 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.193 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.193 INFO:scrape: 2022-04-07T10:51:54.193 INFO:scrape:Crash: reached maximum tries (50) after waiting for 300 seconds 2022-04-07T10:51:54.194 INFO:scrape:ceph version 17.0.0-11492-g46d4842c (46d4842c29bb1566ea5a5be2b85263044364e053) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f3e9e70cce0] 2: /lib64/libpthread.so.0(+0x1181d) [0x7f3e9e70b81d] 3: __pthread_mutex_lock() 4: (rbd::mirror::NamespaceReplayer::add_peer(rbd::mirror::Peer const&, Context*)+0x18b) [0x55cbd04f27cb] 5: (rbd::mirror::PoolReplayer::add_peer(rbd::mirror::PeerSpec const&, C_GatherBase*)+0x9fb) [0x55cbd048c45b] 6: (rbd::mirror::PoolReplayer::update_peers(std::set > const&)+0x5df) [0x55cbd048fddf] 7: (rbd::mirror::Mirror::update_pool_replayers(std::map >, std::less, std::allocator > > > > const&, std::__cxx11::basic_string, std::allocator > const&)+0x72b) [0x55cbd047c25b] 8: (rbd::mirror::Mirror::run()+0x348) [0x55cbd047cab8] 9: main() 10: __libc_start_main() 11: _start() 2022-04-07T10:51:54.194 INFO:scrape:1 jobs: ['6779082'] 2022-04-07T10:51:54.195 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.195 INFO:scrape: 2022-04-07T10:51:54.196 INFO:scrape:Failure: "2022-04-06T13:05:57.310287+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-04-07T10:51:54.196 INFO:scrape:1 jobs: ['6779086'] 2022-04-07T10:51:54.196 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.197 INFO:scrape: 2022-04-07T10:51:54.197 INFO:scrape:Dead: b'2022-04-07T10:47:25.894 DEBUG:teuthology.exit:Finished running handlers' 2022-04-07T10:51:54.198 INFO:scrape:ceph version 17.0.0-11492-g46d4842c (46d4842c29bb1566ea5a5be2b85263044364e053) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fe041618ce0] 2: pthread_cond_wait() 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fe0404218f0] 4: (rbd::mirror::LeaderWatcher::shut_down()+0xfb) [0x56068043d5bb] 5: (rbd::mirror::PoolReplayer::shut_down()+0x87) [0x5606803f7307] 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x5606803f7630] 7: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x3a) [0x5606803e5eca] 8: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x25) [0x5606803e5eb5] 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5606803e0af5] 10: main() 11: __libc_start_main() 12: _start() 2022-04-07T10:51:54.198 INFO:scrape:2 jobs: ['6779098', '6779093'] 2022-04-07T10:51:54.199 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.199 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.200 INFO:scrape: 2022-04-07T10:51:54.200 INFO:scrape:Dead: b'2022-04-06T19:29:41.153 DEBUG:teuthology.exit:Finished running handlers' 2022-04-07T10:51:54.200 INFO:scrape:ceph version 17.0.0-11492-g46d4842c (46d4842c29bb1566ea5a5be2b85263044364e053) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f177351dce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f17740eb192] 5: /usr/lib64/ceph/libceph-common.so.2(+0x284355) [0x7f17740eb355] 6: (rbd::mirror::NamespaceReplayer::add_peer(rbd::mirror::Peer const&, Context*)+0x3f2) [0x55ac51bd4a32] 7: (rbd::mirror::PoolReplayer::update_namespace_replayers()+0x451) [0x55ac51b73f61] 8: (rbd::mirror::PoolReplayer::run()+0x9b3) [0x55ac51b75213] 9: (rbd::mirror::PoolReplayer::PoolReplayerThread::entry()+0x11) [0x55ac51b75e01] 10: /lib64/libpthread.so.0(+0x81cf) [0x7f17735131cf] 11: clone() 2022-04-07T10:51:54.201 INFO:scrape:1 jobs: ['6779087'] 2022-04-07T10:51:54.201 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T10:51:54.202 INFO:scrape: 2022-04-07T10:51:54.203 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 9 dead: 3 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [6779081] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:29:25 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779081/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779081/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d9cbd3dbb874e47b2bb09663a3ad57e Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi137 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6779082] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 01:35:33 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779082/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779082/ reached maximum tries (50) after waiting for 300 seconds [6779084] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:35:32 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779084/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779084/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9192b11ad2b44cdabc86d558265442a Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi138 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6779085] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:25:10 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779085/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779085/ sentry: https://sentry.ceph.com/organizations/ceph/?query=927ce421d47f411585dde8503c0f2854 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi189 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6779086] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:16:28 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779086/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779086/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8db110f940214b1fb7a7108594baa660 "2022-04-06T13:05:57.310287+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6779088] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:27:12 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779088/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b113cc3627c3439d8207cf9f328adf2c Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6779090] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:15:42 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779090/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779090/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7df541ed671940a48fadc3fc88bfe3ed reached maximum tries (90) after waiting for 540 seconds [6779091] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:29:15 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779091/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a12349b25ea45268fe7267a65d3fe7f Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi154 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6779095] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 01:01:45 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779095/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779095/ reached maximum tries (50) after waiting for 300 seconds Dead ================================================================= [6779087] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779087/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779087/ hit max job timeout [6779093] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779093/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779093/ hit max job timeout [6779098] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779098/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779098/ hit max job timeout Pass ================================================================= [6779083] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} time: 00:25:08 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779083/ [6779089] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:24:28 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779089/ [6779092] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} time: 00:26:14 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779092/ [6779094] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}} time: 00:25:19 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779094/ [6779096] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:25:08 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779096/ [6779097] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:24:58 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_12:36:13-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779097/