2022-06-17T10:24:40.048 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-17T10:24:40.051 INFO:teuthology.results:seed: '1736' 2022-06-17T11:06:12.308 INFO:root:teuthology version: 1.1.1.dev374+g31a356e0 2022-06-17T11:06:12.339 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-17T12:01:17.512 INFO:teuthology.results:Tests finished! gathering results... 2022-06-17T12:01:24.063 INFO:scrape:Found 46 jobs 2022-06-17T12:01:31.731 INFO:scrape:Found 4 distinct failure reasons 2022-06-17T12:01:31.739 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi149 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-06-17T12:01:31.742 INFO:scrape:6 jobs: ['6884209', '6884212', '6884218', '6884192', '6884234', '6884203'] 2022-06-17T12:01:31.743 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-06-17T12:01:31.744 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-06-17T12:01:31.745 INFO:scrape: 2022-06-17T12:01:31.746 INFO:scrape:Failure: "2022-06-17T10:59:55.452099+0000 mon.a (mon.0) 94 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-06-17T12:01:31.746 INFO:scrape:1 jobs: ['6884215'] 2022-06-17T12:01:31.747 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-06-17T12:01:31.748 INFO:scrape: 2022-06-17T12:01:31.755 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-06-17T12:01:31.756 INFO:scrape:7 jobs: ['6884208', '6884217', '6884237', '6884211', '6884214', '6884194', '6884219'] 2022-06-17T12:01:31.757 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-06-17T12:01:31.757 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-06-17T12:01:31.758 INFO:scrape: 2022-06-17T12:01:31.759 INFO:scrape:Crash: Found coredumps on ubuntu@smithi110.front.sepia.ceph.com 2022-06-17T12:01:31.760 INFO:scrape:ceph version 17.0.0-12901-g23029423 (2302942357d40471993bc52109ae57214f768c40) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f10179dc420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f1017f7cab4] 5: /usr/lib/ceph/libceph-common.so.2(+0x2b9c5e) [0x7f1017f7cc5e] 6: (rbd::mirror::NamespaceReplayer::init_remote_pool_watcher(rbd::mirror::Peer&, Context*)+0x43f) [0x55d414f24c1f] 7: (rbd::mirror::NamespaceReplayer::init_remote_pool_watchers(Context*)+0x1e0) [0x55d414f298b0] 8: (rbd::mirror::NamespaceReplayer::handle_init_local_pool_watcher(int, Context*)+0x51) [0x55d414f29fb1] 9: (Context::complete(int)+0xd) [0x55d414eb438d] 10: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x55d414eb482a] 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x55d415045235] 12: /lib/librados.so.2(+0xdf358) [0x7f1018cf8358] 13: /lib/librados.so.2(+0xe50fa) [0x7f1018cfe0fa] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f10178b6de4] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f10179d0609] 16: clone() 2022-06-17T12:01:31.760 INFO:scrape:1 jobs: ['6884222'] 2022-06-17T12:01:31.761 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-06-17T12:01:31.762 INFO:scrape: 2022-06-17T12:01:31.783 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 15 dead: 0 running: 0 waiting: 0 queued: 0 passed: 31 Fail ================================================================= [6884192] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 01:06:27 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884192/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1656a7145f6647e69d5876699445b2ae Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi201 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6884194] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:53:03 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884194/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb11b302da634f83a9493bd109ef73e7 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6884203] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:36:30 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884203/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884203/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a4b69f073814d42862b85114b9583d7 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi109 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6884208] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:17:16 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884208/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884208/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e6daad3d0f64273a0cdf1f6944d4bc0 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6884209] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:35:08 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884209/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884209/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7460a83777054dae9adb29c86ad2b6f1 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi149 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6884211] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:40:20 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884211/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06afbd66a87e42e58cc060c815ccca33 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6884212] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:32:56 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884212/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884212/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea7ae96e00694e22aa0f5f243046c9de Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi139 with status 16: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6884214] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:20:44 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884214/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884214/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a075b88a751245c9bbb74a97c74d8ba7 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6884215] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:19:18 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884215/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea6dc8fd03cd45318236ce8245fbc47a "2022-06-17T10:59:55.452099+0000 mon.a (mon.0) 94 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [6884217] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:18:03 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884217/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88386fd8c15542cfbd651a9a2d1ba817 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6884218] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 01:02:58 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884218/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884218/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f7184a13df440528e8c8967a538228c Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6884219] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:19:31 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884219/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=05aae65e189e44a0bdc87aeb6459ed80 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6884222] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:37:50 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884222/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884222/ Found coredumps on ubuntu@smithi110.front.sepia.ceph.com [6884234] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:34:11 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884234/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884234/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3cb9b824d70b478ab9f86b79b6299590 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6884237] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:16:57 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884237/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884237/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5db71a5af6954c198ff7d0e0e7f4dad3 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1627cdc5a79a3e725b98aa8044dcd82c987dd0cd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' Pass ================================================================= [6884193] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:40:47 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884193/ [6884195] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} time: 00:30:49 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884195/ [6884196] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} time: 00:44:01 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884196/ [6884197] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:43:21 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884197/ [6884198] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}} time: 00:32:57 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884198/ [6884199] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:43:28 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884199/ [6884200] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{centos_8}} time: 01:01:52 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884200/ [6884201] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} time: 00:41:18 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884201/ [6884202] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:38:24 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884202/ [6884204] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:32:24 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884204/ [6884205] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} time: 00:42:50 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884205/ [6884206] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{ubuntu_latest}} time: 01:00:24 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884206/ [6884207] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} time: 00:40:48 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884207/ [6884210] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{centos_8}} time: 00:31:07 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884210/ [6884213] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} time: 00:38:23 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884213/ [6884216] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} time: 00:31:53 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884216/ [6884220] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}} time: 00:55:31 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884220/ [6884221] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} time: 00:30:14 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884221/ [6884223] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} time: 00:53:42 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884223/ [6884224] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} time: 00:39:39 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884224/ [6884225] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}} time: 00:55:10 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884225/ [6884226] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}} time: 00:31:03 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884226/ [6884227] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} time: 00:37:35 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884227/ [6884228] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}} time: 00:58:27 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884228/ [6884229] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8}} time: 00:39:43 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884229/ [6884230] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{ubuntu_latest}} time: 00:37:43 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884230/ [6884231] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}} time: 00:55:02 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884231/ [6884232] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} time: 00:31:16 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884232/ [6884233] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}} time: 00:36:27 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884233/ [6884235] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} time: 00:37:04 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884235/ [6884236] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest}} time: 00:28:51 info: http://pulpito.ceph.com/mrfreezeex-2022-06-17_10:24:01-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6884236/