2022-03-16T10:55:34.782 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-16T10:55:34.784 INFO:teuthology.results:seed: '702' 2022-03-16T10:56:24.342 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-16T10:56:24.348 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-16T12:08:29.083 INFO:teuthology.results:Tests finished! gathering results... 2022-03-16T12:08:29.830 INFO:scrape:Found 19 jobs 2022-03-16T12:08:37.825 INFO:scrape:Found 5 distinct failure reasons 2022-03-16T12:08:37.827 INFO:scrape:Crash: Found coredumps on ubuntu@smithi102.front.sepia.ceph.com 2022-03-16T12:08:37.827 INFO:scrape:ceph version 17.0.0-11089-gf0baaf36 (f0baaf3694eacaeddea5f2758fc58361770a25c5) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f434459e3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f4344aff834] 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f4344aff9de] 6: (rbd::mirror::NamespaceReplayer::init_remote_pool_watcher(rbd::mirror::Peer&, Context*)+0x3f9) [0x560fe387e119] 7: (rbd::mirror::NamespaceReplayer::init_remote_pool_watchers(Context*)+0x1e0) [0x560fe3882d30] 8: (rbd::mirror::NamespaceReplayer::handle_init_local_pool_watcher(int, Context*)+0x51) [0x560fe3883431] 9: (Context::complete(int)+0xd) [0x560fe380d9ad] 10: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x560fe380de4a] 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x560fe399c985] 12: /lib/librados.so.2(+0xdf138) [0x7f4345802138] 13: /lib/librados.so.2(+0xe4dda) [0x7f4345807dda] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f4344477de4] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f4344592609] 16: clone() 2022-03-16T12:08:37.827 INFO:scrape:2 jobs: ['6739099', '6739100'] 2022-03-16T12:08:37.828 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T12:08:37.828 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T12:08:37.829 INFO:scrape: 2022-03-16T12:08:37.829 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi166 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-03-16T12:08:37.830 INFO:scrape:10 jobs: ['6739091', '6739105', '6739088', '6739093', '6739095', '6739089', '6739102', '6739097', '6739096', '6739104'] 2022-03-16T12:08:37.830 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T12:08:37.830 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T12:08:37.831 INFO:scrape: 2022-03-16T12:08:37.831 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-03-16T12:08:37.832 INFO:scrape:4 jobs: ['6739087', '6739101', '6739094', '6739098'] 2022-03-16T12:08:37.832 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-16T12:08:37.833 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-stupid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-16T12:08:37.833 INFO:scrape: 2022-03-16T12:08:37.834 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-03-16T12:08:37.834 INFO:scrape:1 jobs: ['6739090'] 2022-03-16T12:08:37.834 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-16T12:08:37.835 INFO:scrape: 2022-03-16T12:08:37.835 INFO:scrape:Crash: Found coredumps on ubuntu@smithi195.front.sepia.ceph.com 2022-03-16T12:08:37.836 INFO:scrape:ceph version 17.0.0-11089-gf0baaf36 (f0baaf3694eacaeddea5f2758fc58361770a25c5) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fefa427f3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7fefa47e0834] 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7fefa47e09de] 6: (rbd::mirror::NamespaceReplayer::init_remote_pool_watcher(rbd::mirror::Peer&, Context*)+0x3f9) [0x55abea716119] 7: (rbd::mirror::NamespaceReplayer::init_remote_pool_watchers(Context*)+0x1e0) [0x55abea71ad30] 8: (rbd::mirror::NamespaceReplayer::handle_init_local_pool_watcher(int, Context*)+0x51) [0x55abea71b431] 9: (Context::complete(int)+0xd) [0x55abea6a59ad] 10: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x55abea6a5e4a] 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x55abea834985] 12: /lib/librados.so.2(+0xdf138) [0x7fefa54e3138] 13: /lib/librados.so.2(+0xe4dda) [0x7fefa54e8dda] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7fefa4158de4] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fefa4273609] 16: clone() 2022-03-16T12:08:37.836 INFO:scrape:1 jobs: ['6739103'] 2022-03-16T12:08:37.836 INFO:scrape:suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-16T12:08:37.837 INFO:scrape: 2022-03-16T12:08:37.838 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 18 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6739087] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:41:38 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739087/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739087/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96b0a3b9aecf4f87b999776463b5faca Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739088] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:30:12 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739088/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b00ff59330ae4ee68c2f41ed8006cdfa Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739089] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:49:51 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739089/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739089/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c61ab6c5699425e8203eb2cb5e66ac4 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739090] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:18:37 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739090/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739090/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffd7c29c926b49468b86ae956ded6744 reached maximum tries (90) after waiting for 540 seconds [6739091] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:26:29 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739091/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e587b562841a434bb30630cc2812e952 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi166 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739093] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:28:03 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739093/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739093/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b6e7f8b1e0cf40b5ba927063ee415b93 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi141 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739094] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:32:16 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739094/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739094/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17fd72f96cd54e4ba3eab1583bb4c76e Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi142 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739095] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:49:23 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739095/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739095/ sentry: https://sentry.ceph.com/organizations/ceph/?query=068e9f6726d94263b36ad9b7bdfdb084 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739096] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:29:53 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739096/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739096/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e14c1306838c4d7482ed118ff95aea73 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739097] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:50:53 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739097/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739097/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b7c9e1ca37c4f6a9c72da9fb67aced8 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739098] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:41:01 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739098/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739098/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39eeaa269a2349a689c85d2790e5c54f Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi185 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739099] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:49:24 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739099/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739099/ Found coredumps on ubuntu@smithi102.front.sepia.ceph.com [6739100] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:58:08 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739100/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739100/ Found coredumps on ubuntu@smithi167.front.sepia.ceph.com [6739101] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:40:41 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739101/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=663bc351bc394ee68fc605540bcc1ac8 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi204 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6739102] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:49:40 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739102/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739102/ sentry: https://sentry.ceph.com/organizations/ceph/?query=26dd2afd54804d4fa0d419b0056a0cf3 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739103] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:55:38 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739103/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739103/ Found coredumps on ubuntu@smithi195.front.sepia.ceph.com [6739104] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:56:14 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739104/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739104/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed5b69e455334c9fac18252706716f7f Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6739105] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:30:35 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739105/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b8e163c815c40ad8f7f66157b883136 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4b7501e6b05416cdec8a3a8672027d6e7f6fc7ef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' Pass ================================================================= [6739092] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} time: 00:26:12 info: http://pulpito.ceph.com/mrfreezeex-2022-03-16_10:55:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739092/