Found 189 jobs Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Not-understood crash indication 2022-02-11T11:23:47.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi155.stderr:daemon-helper: command crashed with signal 11 Not-understood crash indication Found 10 distinct failure reasons Failure: 18 jobs: ['6676506', '6676527', '6676573', '6676584', '6676457', '6676440', '6676510', '6676530', '6676563', '6676461', '6676452', '6676577', '6676554', '6676471', '6676550', '6676489', '6676407', '6676497'] suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}'] suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-journal-workunit-multipeer}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] Failure: "2022-02-11T12:51:49.551056+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: Degraded data redundancy: 1/3008 objects degraded (0.033%), 1 pg degraded (PG_DEGRADED)" in cluster log 3 jobs: ['6676535', '6676412', '6676495'] suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}', 'supported-random-distro$/{centos_8}'] suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zstd', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-snapshot-workunit-fast-diff-multipeer}'] Failure: Command failed (workunit test rbd/rbd_mirror_snapshot.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d9c24494798767914b7df5d93356fed1935d23b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_CONFIG_KEY=1 RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_snapshot.sh' 4 jobs: ['6676528', '6676515', '6676455', '6676410'] suites intersection: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'openstack}', 'rbd/mirror/{base/install'] suites union: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-workunit-minimum}'] Failure: No module named 'tasks.rbd_fsx' 4 jobs: ['6676469', '6676487', '6676473', '6676472'] suites intersection: ['cluster/{2-node', 'msgr-failures/few', 'openstack}'] suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'workloads/rbd-mirror-snapshot-workunit-exclusive-lock}'] Crash: Found coredumps on ubuntu@smithi173.front.sepia.ceph.com ceph version 17.0.0-10676-g9d9c2449 (9d9c24494798767914b7df5d93356fed1935d23b) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7fc3f0ff9c20] 2: gsignal() 3: abort() 4: rbd-mirror(+0x55dd98) [0x55782ce6ed98] 5: (librbd::util::detail::C_AsyncCallback::finish(int)+0x85) [0x55782ce78145] 6: (Context::complete(int)+0xd) [0x55782ce73b7d] 7: rbd-mirror(+0x78bbf4) [0x55782d09cbf4] 8: (Context::complete(int)+0xd) [0x55782ce73b7d] 9: (librbd::journal::util::C_DecodeTags::complete(int)+0x24) [0x55782d09d414] 10: rbd-mirror(+0xd29640) [0x55782d63a640] 11: /lib64/librados.so.2(+0xac046) [0x7fc3f3112046] 12: /lib64/librados.so.2(+0xc6dc5) [0x7fc3f312cdc5] 13: /lib64/librados.so.2(+0xc8eb8) [0x7fc3f312eeb8] 14: /lib64/librados.so.2(+0xce25a) [0x7fc3f313425a] 15: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fc3efe08ba3] 16: /lib64/libpthread.so.0(+0x817f) [0x7fc3f0fef17f] 17: clone() 1 jobs: ['6676450'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-journal-stress-workunit}'] Failure: reached maximum tries (90) after waiting for 540 seconds 1 jobs: ['6676517'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-journal-workunit-multipeer}'] Failure: Command failed on smithi037 with status 1: 'sudo yum -y install cephfs-java' 1 jobs: ['6676478'] suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-stress-workunit-min-compat-client-octopus}'] Crash: "2022-02-11T13:55:26.587012+0000 mon.a (mon.0) 1818 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log ceph version 17.0.0-10676-g9d9c2449 (9d9c24494798767914b7df5d93356fed1935d23b) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f585e8013c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x5589b453ef07] 5: (KernelDevice::_aio_thread()+0xe19) [0x5589b5216099] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5589b5220dc1] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f585e7f5609] 8: clone() 1 jobs: ['6676587'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] Failure: Found coredumps on ubuntu@smithi155.front.sepia.ceph.com 1 jobs: ['6676405'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-journal-stress-workunit}'] Crash: Found coredumps on ubuntu@smithi065.front.sepia.ceph.com ceph version 17.0.0-10676-g9d9c2449 (9d9c24494798767914b7df5d93356fed1935d23b) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f81c7180c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f81c7d4d372] 5: /usr/lib64/ceph/libceph-common.so.2(+0x283535) [0x7f81c7d4d535] 6: (rbd::mirror::NamespaceReplayer::shut_down_remote_status_updater(rbd::mirror::Peer&, Context*)+0x1ab) [0x55e3d2935eab] 7: (rbd::mirror::NamespaceReplayer::handle_remove_peer(rbd::mirror::Peer const&, int, Context*)+0x203) [0x55e3d2936103] 8: (Context::complete(int)+0xd) [0x55e3d28c4b7d] 9: (boost::asio::detail::completion_handler, 0u> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x5a) [0x55e3d28c500a] 10: (boost::asio::detail::strand_service::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x85) [0x55e3d2a95ad5] 11: /lib64/librados.so.2(+0xc8eb8) [0x7f81c92b5eb8] 12: /lib64/librados.so.2(+0xce25a) [0x7f81c92bb25a] 13: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f81c5f8fba3] 14: /lib64/libpthread.so.0(+0x817f) [0x7f81c717617f] 15: clone() 1 jobs: ['6676437'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'policy/none', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/rbd-mirror-journal-stress-workunit}']