2022-03-15T11:25:06.407 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-15T11:25:06.410 INFO:teuthology.results:seed: '2491' 2022-03-15T11:43:13.006 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-15T11:43:13.076 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-15T12:42:17.550 INFO:teuthology.results:Tests finished! gathering results... 2022-03-15T12:42:18.798 INFO:scrape:Found 18 jobs 2022-03-15T12:42:25.401 INFO:scrape:Found 4 distinct failure reasons 2022-03-15T12:42:25.402 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-03-15T12:42:25.402 INFO:scrape:5 jobs: ['6738228', '6738238', '6738233', '6738241', '6738244'] 2022-03-15T12:42:25.403 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-15T12:42:25.404 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-15T12:42:25.404 INFO:scrape: 2022-03-15T12:42:25.405 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-03-15T12:42:25.405 INFO:scrape:7 jobs: ['6738227', '6738234', '6738230', '6738240', '6738236', '6738242', '6738229'] 2022-03-15T12:42:25.406 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T12:42:25.406 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T12:42:25.407 INFO:scrape: 2022-03-15T12:42:25.408 INFO:scrape:Crash: Found coredumps on ubuntu@smithi148.front.sepia.ceph.com 2022-03-15T12:42:25.408 INFO:scrape:ceph version 17.0.0-11089-gf0baaf36 (f0baaf3694eacaeddea5f2758fc58361770a25c5) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f1ad58c4c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f1ad6492262] 5: /usr/lib64/ceph/libceph-common.so.2(+0x284425) [0x7f1ad6492425] 6: (rbd::mirror::NamespaceReplayer::init_remote_pool_watcher(rbd::mirror::Peer&, Context*)+0x40c) [0x557f74ff7f0c] 7: (rbd::mirror::NamespaceReplayer::init_remote_pool_watchers(Context*)+0x299) [0x557f74ffb9b9] 8: (rbd::mirror::NamespaceReplayer::handle_init_local_pool_watcher(int, Context*)+0x51) [0x557f74ffbf71] 9: (Context::complete(int)+0xd) [0x557f74f86fdd] 10: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x557f74f8746a] 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x557f75158ad5] 12: /lib64/librados.so.2(+0xc8ee8) [0x7f1ad79fbee8] 13: /lib64/librados.so.2(+0xce28a) [0x7f1ad7a0128a] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f1ad46d3ba3] 15: /lib64/libpthread.so.0(+0x817f) [0x7f1ad58ba17f] 16: clone() 2022-03-15T12:42:25.409 INFO:scrape:2 jobs: ['6738239', '6738237'] 2022-03-15T12:42:25.409 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T12:42:25.410 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T12:42:25.410 INFO:scrape: 2022-03-15T12:42:25.410 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-03-15T12:42:25.411 INFO:scrape:2 jobs: ['6738243', '6738232'] 2022-03-15T12:42:25.412 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T12:42:25.412 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-15T12:42:25.413 INFO:scrape: 2022-03-15T12:42:25.414 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 16 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [6738227] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:24:13 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738227/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738227/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb6162fccd3c4d739588be7a92045958 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738228] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:43:48 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738228/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738228/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cb52b18285694519905623ff3f6d8560 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738229] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:43:21 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738229/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738229/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a003bdc442d4d41a6921a4bc64c8472 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi197 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738230] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:44:59 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738230/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738230/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b107c1288f7a44cfb067c183901d449d Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738232] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:18:56 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738232/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738232/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b4447770d6a4f5b87a90ffc5b4ea5a8 reached maximum tries (90) after waiting for 540 seconds [6738233] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:44:02 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738233/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738233/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1b4a6a46ab1493ab5cc0ce603694d41 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738234] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:24:35 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738234/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738234/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6cf6b09873984420b93ac04e64760e1b Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi196 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738236] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:25:02 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738236/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738236/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fcf4ef843f07407cb3e28e78a7e73d9c Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738237] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:54:57 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738237/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738237/ Found coredumps on ubuntu@smithi071.front.sepia.ceph.com [6738238] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:26:53 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738238/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738238/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5cb7d6f7ae54b45b553ff4b06df8fee Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738239] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:51:31 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738239/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738239/ Found coredumps on ubuntu@smithi148.front.sepia.ceph.com [6738240] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:24:40 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738240/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738240/ sentry: https://sentry.ceph.com/organizations/ceph/?query=003d0268ecaa45808499dd738152d2d1 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi153 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738241] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:43:26 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738241/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738241/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2efb311a544f4535ae73bdc974152e61 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6738242] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:24:50 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738242/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738242/ sentry: https://sentry.ceph.com/organizations/ceph/?query=947da1b5db2846edb29a0d28762b6ea4 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6738243] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:18:52 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738243/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738243/ sentry: https://sentry.ceph.com/organizations/ceph/?query=262759284079409fadb37ff900e1e699 reached maximum tries (90) after waiting for 540 seconds [6738244] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:44:15 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738244/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738244/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1aef8300b54442890343a0edb40b829 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi179 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4f90a5d406228a99c73b8ecd0c162c9e57084aa0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' Pass ================================================================= [6738231] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{centos_8}} time: 00:27:34 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738231/ [6738235] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8}} time: 00:28:06 info: http://pulpito.ceph.com/mrfreezeex-2022-03-15_11:24:47-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6738235/