2022-03-17T21:58:27.805 INFO:root:teuthology version: 1.1.0-cd8299ff 2022-03-17T21:58:27.807 INFO:teuthology.results:seed: '7234' 2022-03-17T22:13:40.370 INFO:root:teuthology version: 1.1.0-cd8299ff 2022-03-17T22:13:40.407 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-17T23:11:44.645 INFO:teuthology.results:Tests finished! gathering results... 2022-03-17T23:11:45.349 INFO:scrape:Found 15 jobs 2022-03-17T23:11:50.429 INFO:scrape:Found 3 distinct failure reasons 2022-03-17T23:11:50.430 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi185 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 2022-03-17T23:11:50.430 INFO:scrape:7 jobs: ['6742821', '6742824', '6742814', '6742823', '6742811', '6742819', '6742825'] 2022-03-17T23:11:50.431 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-17T23:11:50.432 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-17T23:11:50.432 INFO:scrape: 2022-03-17T23:11:50.433 INFO:scrape:Crash: Found coredumps on ubuntu@smithi183.front.sepia.ceph.com 2022-03-17T23:11:50.433 INFO:scrape:ceph version 17.0.0-11156-gbd53a04b (bd53a04b58fa92a82a5f1702df59a11febbfceae) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fddab8123c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7fddabd73834] 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7fddabd739de] 6: (rbd::mirror::NamespaceReplayer::init_remote_pool_watcher(rbd::mirror::Peer&, Context*)+0x3f9) [0x55ad3e65ca39] 7: (rbd::mirror::NamespaceReplayer::init_remote_pool_watchers(Context*)+0x1e0) [0x55ad3e661650] 8: (rbd::mirror::NamespaceReplayer::handle_init_local_pool_watcher(int, Context*)+0x51) [0x55ad3e661d51] 9: (Context::complete(int)+0xd) [0x55ad3e5ec2cd] 10: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x55ad3e5ec76a] 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x55ad3e77cef5] 12: /lib/librados.so.2(+0xdf138) [0x7fddaca77138] 13: /lib/librados.so.2(+0xe4dda) [0x7fddaca7cdda] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7fddab6ebde4] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fddab806609] 16: clone() 2022-03-17T23:11:50.434 INFO:scrape:3 jobs: ['6742816', '6742813', '6742817'] 2022-03-17T23:11:50.434 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-17T23:11:50.435 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] 2022-03-17T23:11:50.435 INFO:scrape: 2022-03-17T23:11:50.436 INFO:scrape:Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2022-03-17T23:11:50.436 INFO:scrape:4 jobs: ['6742815', '6742818', '6742822', '6742812'] 2022-03-17T23:11:50.437 INFO:scrape:suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-17T23:11:50.437 INFO:scrape:suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'objectstore/filestore-xfs', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] 2022-03-17T23:11:50.438 INFO:scrape: 2022-03-17T23:11:50.439 INFO:teuthology.results:Sending results to arthur.outhenin-chalandre@cern.ch: Test Run: mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/ failed: 14 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6742811] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:52:54 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742811/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742811/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd8c40831d274576a61913f45ebbe0e9 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6742812] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:31:28 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742812/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742812/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ead62aa07b4d46bcb6c98ac01ab08537 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6742813] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:50:51 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742813/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742813/ Found coredumps on ubuntu@smithi186.front.sepia.ceph.com [6742814] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:55:39 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742814/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742814/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63eba993d4ec451a99aa20ad11486c6d Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi204 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6742815] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:32:57 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742815/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742815/ sentry: https://sentry.ceph.com/organizations/ceph/?query=34dd4c93ff7b4d9785a7fa2eeaf85417 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6742816] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:51:08 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742816/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742816/ Found coredumps on ubuntu@smithi183.front.sepia.ceph.com [6742817] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:53:29 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742817/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742817/ Found coredumps on ubuntu@smithi134.front.sepia.ceph.com [6742818] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/filestore-xfs supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:33:36 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742818/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742818/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eb68bf5081a342158011bc23547b5b48 Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi109 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6742819] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:27:35 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742819/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d5298c3d53e4c6a81949738829e7be7 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6742821] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:31:01 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742821/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6469deb11cb54774b7b0fcc6de2524c4 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi185 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6742822] rbd/mirror/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{ubuntu_latest}} ----------------------------------------------------------------- time: 00:33:20 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742822/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e1f66e0c09e4820bb02e5b5834a990d Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' [6742823] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:50:51 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742823/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f00971643a94612a16ee83f7f41f90c Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6742824] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:31:13 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742824/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e74a8553e2840b39203f63283da1a1c Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi190 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' [6742825] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:29:32 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742825/ log: http://qa-proxy.ceph.com/teuthology/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4146ea5f6cd4002993d3456151a5514 Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi197 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bd53a04b58fa92a82a5f1702df59a11febbfceae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' Pass ================================================================= [6742820] rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}} time: 00:52:36 info: http://pulpito.ceph.com/mrfreezeex-2022-03-17_17:31:27-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6742820/