2022-04-07T04:12:15.558 INFO:root:teuthology version: 1.1.0-3dd79763 2022-04-07T04:12:15.559 INFO:teuthology.results:seed: '1112' 2022-04-07T07:54:59.963 INFO:root:teuthology version: 1.1.0-3dd79763 2022-04-07T07:55:00.025 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-07T11:42:16.168 INFO:teuthology.results:Tests finished! gathering results... 2022-04-07T11:42:18.015 INFO:scrape:Found 21 jobs 2022-04-07T11:42:31.042 INFO:scrape:Found 7 distinct failure reasons 2022-04-07T11:42:31.043 INFO:scrape:Dead: b'2022-04-07T10:54:39.209 DEBUG:teuthology.exit:Finished running handlers' 2022-04-07T11:42:31.044 INFO:scrape:ceph version 17.0.0-11492-g46d4842c (46d4842c29bb1566ea5a5be2b85263044364e053) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7fbe4aad73c0] 2: pthread_cond_wait() 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fbe4a9abe30] 4: (rbd::mirror::LeaderWatcher::shut_down()+0x10b) [0x55dc21941f8b] 5: (rbd::mirror::PoolReplayer::shut_down()+0x99) [0x55dc218fb7d9] 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55dc218fbba0] 7: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x3e) [0x55dc218e830e] 8: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x28) [0x55dc218e82f8] 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x55dc218e12b5] 10: main() 11: __libc_start_main() 12: _start() 2022-04-07T11:42:31.045 INFO:scrape:5 jobs: ['6779245', '6779252', '6779248', '6779258', '6779263'] 2022-04-07T11:42:31.046 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.046 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.047 INFO:scrape: 2022-04-07T11:42:31.048 INFO:scrape:Failure: "2022-04-07T04:50:29.537944+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-04-07T11:42:31.049 INFO:scrape:2 jobs: ['6779254', '6779253'] 2022-04-07T11:42:31.050 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.051 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-stupid', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.052 INFO:scrape: 2022-04-07T11:42:31.053 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-04-07T11:42:31.053 INFO:scrape:2 jobs: ['6779261', '6779250'] 2022-04-07T11:42:31.054 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.055 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.056 INFO:scrape: 2022-04-07T11:42:31.057 INFO:scrape:Failure: Command failed on smithi165 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 46d4842c29bb1566ea5a5be2b85263044364e053' 2022-04-07T11:42:31.058 INFO:scrape:2 jobs: ['6779265', '6779264'] 2022-04-07T11:42:31.059 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'mirror', 'msgr-failures/few', 'roles}', 'supported-random-distro$/{ubuntu_latest}}'] 2022-04-07T11:42:31.060 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-04-07T11:42:31.061 INFO:scrape: 2022-04-07T11:42:31.061 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2022-04-07T11:42:31.062 INFO:scrape:1 jobs: ['6779249'] 2022-04-07T11:42:31.063 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.064 INFO:scrape: 2022-04-07T11:42:31.065 INFO:scrape:Crash: reached maximum tries (50) after waiting for 300 seconds 2022-04-07T11:42:31.066 INFO:scrape:ceph version 17.0.0-11492-g46d4842c (46d4842c29bb1566ea5a5be2b85263044364e053) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7f58c1cf23c0] 2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x13110) [0x7f58c1cf1110] 3: pthread_mutex_lock() 4: (rbd::mirror::NamespaceReplayer::add_peer(rbd::mirror::Peer const&, Context*)+0x63) [0x55560e51e1e3] 5: (rbd::mirror::PoolReplayer::add_peer(rbd::mirror::PeerSpec const&, C_GatherBase*)+0x8c1) [0x55560e4b9cc1] 6: (rbd::mirror::PoolReplayer::update_peers(std::set > const&)+0x65b) [0x55560e4bd3cb] 7: (rbd::mirror::Mirror::update_pool_replayers(std::map >, std::less, std::allocator > > > > const&, std::__cxx11::basic_string, std::allocator > const&)+0x520) [0x55560e4a69d0] 8: (rbd::mirror::Mirror::run()+0x214) [0x55560e4a7654] 9: main() 10: __libc_start_main() 11: _start() 2022-04-07T11:42:31.067 INFO:scrape:1 jobs: ['6779259'] 2022-04-07T11:42:31.067 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.068 INFO:scrape: 2022-04-07T11:42:31.069 INFO:scrape:Crash: reached maximum tries (50) after waiting for 300 seconds 2022-04-07T11:42:31.070 INFO:scrape:ceph version 17.0.0-11492-g46d4842c (46d4842c29bb1566ea5a5be2b85263044364e053) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f3587f44ce0] 2: pthread_cond_wait() 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7f3586d4d8f0] 4: (rbd::mirror::LeaderWatcher::shut_down()+0xfb) [0x5617bcb195bb] 5: (rbd::mirror::PoolReplayer::shut_down()+0x87) [0x5617bcad3307] 6: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x5617bcad3630] 7: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x3a) [0x5617bcac1eca] 8: (std::_Rb_tree, std::default_delete > > >, std::_Select1st, std::default_delete > > > >, std::less, std::allocator, std::default_delete > > > > >::_M_erase(std::_Rb_tree_node, std::default_delete > > > >*)+0x25) [0x5617bcac1eb5] 9: (rbd::mirror::Mirror::~Mirror()+0xc5) [0x5617bcabcaf5] 10: main() 11: __libc_start_main() 12: _start() 2022-04-07T11:42:31.071 INFO:scrape:1 jobs: ['6779256'] 2022-04-07T11:42:31.071 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/filestore-xfs', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-04-07T11:42:31.072 INFO:scrape: 2022-04-07T11:42:31.074 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 9 dead: 5 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [6779249] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 01:11:23 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779249/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779249/ reached maximum tries (50) after waiting for 300 seconds [6779250] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:59:42 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779250/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779250/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5c8d564783e46c29cae97be9c010d3e Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6779253] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:19:41 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779253/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779253/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2986c13cf58649c28f71f52f1d5a8ef5 "2022-04-07T04:50:16.938144+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6779254] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:19:19 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779254/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779254/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d349456c0f4461fabe15c240a4ad5e4 "2022-04-07T04:50:29.537944+0000 mon.a (mon.0) 95 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6779256] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 01:22:32 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779256/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779256/ reached maximum tries (50) after waiting for 300 seconds [6779259] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 01:12:50 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779259/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779259/ reached maximum tries (50) after waiting for 300 seconds [6779261] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:30:51 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779261/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779261/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc95180e32a247f4b43c848215527015 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi176 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=46d4842c29bb1566ea5a5be2b85263044364e053 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6779264] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:44 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779264/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779264/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67c667821bd34893b8a96bd7290d9803 Command failed on smithi103 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 46d4842c29bb1566ea5a5be2b85263044364e053' [6779265] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:10:55 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779265/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779265/ sentry: https://sentry.ceph.com/organizations/ceph/?query=474d291fd32d4eeb8ed6b9a0d7bbc5bf Command failed on smithi165 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout 46d4842c29bb1566ea5a5be2b85263044364e053' Dead ================================================================= [6779245] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779245/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779245/ hit max job timeout [6779248] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779248/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779248/ hit max job timeout [6779252] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779252/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779252/ hit max job timeout [6779258] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779258/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779258/ hit max job timeout [6779263] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779263/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779263/ hit max job timeout Pass ================================================================= [6779246] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}} time: 00:25:33 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779246/ [6779247] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}} time: 00:54:03 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779247/ [6779251] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8}} time: 00:27:24 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779251/ [6779255] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:25:02 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779255/ [6779257] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} time: 00:26:55 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779257/ [6779260] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} time: 00:26:33 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779260/ [6779262] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:26:12 info: http://pulpito.ceph.com/mrfreezeex-2022-04-06_14:00:28-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6779262/