Found 21 jobs Missing teuthology log /home/teuthworker/archive/mrfreezeex-2022-03-16_15:04:18-rbd-wip-mrfreezeex-multipeer-distro-default-smithi/6739173/teuthology.log Found 7 distinct failure reasons Crash: Found coredumps on ubuntu@smithi165.front.sepia.ceph.com ceph version 17.0.0-11089-gf0baaf36 (f0baaf3694eacaeddea5f2758fc58361770a25c5) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fa69ae5f3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7fa69b3c0834] 5: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7fa69b3c09de] 6: (rbd::mirror::NamespaceReplayer::init_remote_pool_watcher(rbd::mirror::Peer&, Context*)+0x3f9) [0x55eb4c3d7119] 7: (rbd::mirror::NamespaceReplayer::init_remote_pool_watchers(Context*)+0x1e0) [0x55eb4c3dbd30] 8: (rbd::mirror::NamespaceReplayer::handle_init_local_pool_watcher(int, Context*)+0x51) [0x55eb4c3dc431] 9: (Context::complete(int)+0xd) [0x55eb4c3669ad] 10: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x55eb4c366e4a] 11: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x55eb4c4f5985] 12: /lib/librados.so.2(+0xdf138) [0x7fa69c0c3138] 13: /lib/librados.so.2(+0xe4dda) [0x7fa69c0c8dda] 14: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7fa69ad38de4] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fa69ae53609] 16: clone() 2 jobs: ['6739167', '6739177'] suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] Failure: Command failed on smithi110 with status 1: 'sudo yum install -y kernel' 2 jobs: ['6739172', '6739170'] suites intersection: ['cluster/3-nodes/{base/{install', 'mirror', 'msgr-failures/few', 'roles}', 'supported-random-distro$/{centos_8}}'] suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] Failure: SSH connection to smithi052 was lost: 'sudo apt-get update' 2 jobs: ['6739171', '6739181'] suites intersection: ['cluster/3-nodes/{base/{install', 'mirror', 'msgr-failures/few', 'roles}'] suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-hybrid', 'policy/none', 'rbd/mirror-thrash/{base/{openstack}', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] Failure: Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=279562229535bb021fb62a241d21b6b9da2fe366 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh' 8 jobs: ['6739163', '6739168', '6739174', '6739165', '6739179', '6739180', '6739176', '6739169'] suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] suites union: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'policy/none', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] Dead: None 2 jobs: ['6739173', '6739183'] Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=279562229535bb021fb62a241d21b6b9da2fe366 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 2 jobs: ['6739175', '6739178'] suites intersection: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'rbd/mirror/{base/{openstack}', 'roles}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] suites union: ['cluster/3-nodes/{base/{install', 'extra/{mirror-extra}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-stupid', 'rbd/mirror/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'supported-random-distro$/{ubuntu_latest}}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] Failure: machine smithi017.front.sepia.ceph.com is locked by scheduled_ksirivad@teuthology, not scheduled_mrfreezeex@teuthology 1 jobs: ['6739182'] suites: ['cluster/3-nodes/{base/{install', 'extra/{rbd-mirror}', 'mirror', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'policy/simple', 'rbd/mirror-thrash/{base/{openstack}', 'roles}', 'supported-random-distro$/{centos_8}}', 'workloads/rbd-mirror-journal-workunit-multipeer}']