2022-03-16T15:14:17.693 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-16T15:14:17.695 INFO:teuthology.results:seed: '9173' 2022-03-16T15:15:08.307 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-16T15:15:08.314 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-16T16:20:12.645 INFO:teuthology.results:Tests finished! gathering results... 2022-03-16T16:20:13.220 INFO:scrape:Found 15 jobs 2022-03-16T16:20:18.780 INFO:scrape:Found 3 distinct failure reasons 2022-03-16T16:20:18.781 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi170 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-03-16T16:20:18.782 INFO:scrape:6 jobs: ['6739200', '6739202', '6739189', '6739194', '6739193', '6739190'] 2022-03-16T16:20:18.782 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-16T16:20:18.783 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-16T16:20:18.783 INFO:scrape: 2022-03-16T16:20:18.784 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-03-16T16:20:18.784 INFO:scrape:3 jobs: ['6739196', '6739192', '6739199'] 2022-03-16T16:20:18.785 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T16:20:18.785 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T16:20:18.785 INFO:scrape: 2022-03-16T16:20:18.786 INFO:scrape:Crash: Found coredumps on ubuntu@smithi070.front.sepia.ceph.com 2022-03-16T16:20:18.786 INFO:scrape:ceph version 17.0.0-11089-gf0baaf36 (f0baaf3694eacaeddea5f2758fc58361770a25c5) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f3f43cb33c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f3f44214834] 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f3f442149de] 6: (rbd::mirror::NamespaceReplayer::init_remote_pool_watcher(rbd::mirror::Peer&, Context*)+0x3f9) [0x5644d41a9119] 7: (rbd::mirror::NamespaceReplayer::init_remote_pool_watchers(Context*)+0x1e0) [0x5644d41add30] 8: (rbd::mirror::NamespaceReplayer::handle_init_local_pool_watcher(int, Context*)+0x51) [0x5644d41ae431] 9: (Context::complete(int)+0xd) [0x5644d41389ad] 10: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x5644d4138e4a] 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x5644d42c7985] 12: /lib/librados.so.2(+0xdf138) [0x7f3f44f17138] 13: /lib/librados.so.2(+0xe4dda) [0x7f3f44f1cdda] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f3f43b8cde4] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f3f43ca7609] 16: clone() 2022-03-16T16:20:18.787 INFO:scrape:5 jobs: ['6739201', '6739197', '6739198', '6739195', '6739191'] 2022-03-16T16:20:18.787 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T16:20:18.788 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-stupid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T16:20:18.788 INFO:scrape: 2022-03-16T16:20:18.789 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 14 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6739189] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:39:39 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739189/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abf5328bbbdb43fdb0bc51572f387104 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739190] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:31:06 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739190/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739190/ sentry: https://sentry.ceph.com/organizations/ceph/?query=934f579d2b2c4c4eb38e99772070226e Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739191] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:48:53 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739191/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739191/ Found coredumps on ubuntu@smithi164.front.sepia.ceph.com [6739192] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:23:43 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739192/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739192/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a0073db89d54d999e6dd8a80fcdef90 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739193] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:33:38 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739193/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57654d3a2c5c4253b7568426f35088c1 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi181 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739194] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:37:35 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739194/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7594c94550784b358fea6caeea1f6a34 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739195] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:51:17 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739195/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739195/ Found coredumps on ubuntu@smithi188.front.sepia.ceph.com [6739196] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:29:41 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739196/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739196/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9c2f1581f06426ebc2ffcf3177371cd Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739197] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:50:56 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739197/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739197/ Found coredumps on ubuntu@smithi204.front.sepia.ceph.com [6739198] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:49:56 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739198/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739198/ Found coredumps on ubuntu@smithi183.front.sepia.ceph.com [6739199] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:24:23 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739199/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54fb664d15cb48c0826f08b1e6b2a46f Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739200] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:32:52 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739200/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739200/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ed000ea93c64c77bf959041317a8358 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi170 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739201] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:45:14 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739201/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739201/ Found coredumps on ubuntu@smithi070.front.sepia.ceph.com [6739202] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:34:14 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739202/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1792ebcca19f45abb015cafdf589aaf9 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=39b60fd821df85e0030506fe5a1aff61bb9e8f49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' Pass ================================================================= [6739203] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{ubuntu_latest}} time: 00:50:46 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_15:14:10-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739203/