Found 85 jobs Found 4 distinct failure reasons Failure: "2021-01-02 06:55:03.995762 mon.a (mon.0) 400 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log 2 jobs: ['5741787', '5741703'] suites intersection: ['clusters/3-node', 'conf', 'krbd/fsx/{ceph/ceph', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-3-client}'] suites union: ['clusters/3-node', 'conf', 'krbd/fsx/{ceph/ceph', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'striping/default/{msgr-failures/many', 'tasks/fsx-3-client}'] Crash: Command failed on smithi084 with status 4: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude5n11gt63 -r -- '-g auto -g blockdev -x clone'" ceph version 14.2.16-114-g74f48adff3 (74f48adff35db6f86e9231614da019ef946277a3) nautilus (stable) 1: (()+0x12980) [0x7f5b7b3a1980] 2: (gsignal()+0xc7) [0x7f5b7a053fb7] 3: (abort()+0x141) [0x7f5b7a055921] 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x56030a8a6ae5] 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x185c) [0x56030ab0664c] 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x431f) [0x56030ab0ab2f] 7: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xd08) [0x56030ab0bfd8] 8: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1bf) [0x56030a92a2cf] 9: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x56030abe3eb2] 10: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x56030a9482f5] 11: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x56030af624bc] 12: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x56030af65680] 13: (()+0x76db) [0x7f5b7b3966db] 14: (clone()+0x3f) [0x7f5b7a13671f] 1 jobs: ['5741719'] suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'msgr-failures/many', 'tasks/rbd_xfstests}'] Failure: Command failed on smithi204 with status 4: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludetbai5ktv -r -- '-g auto -g blockdev -x clone'" 1 jobs: ['5741570'] suites: ['conf', 'krbd/singleton/{bluestore-bitmap', 'msgr-failures/few', 'tasks/rbd_xfstests}'] Failure: Command failed (workunit test rbd/krbd_stable_pages_required.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=74f48adff35db6f86e9231614da019ef946277a3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_stable_pages_required.sh' 1 jobs: ['5741582'] suites: ['ceph/ceph', 'clusters/fixed-1', 'conf', 'krbd/wac/sysfs/{bluestore-bitmap', 'tasks/stable_pages_required}']