Found 20 jobs Found 4 distinct failure reasons Failure: Test failure: test_client_abort (tasks.cephfs.test_failover.TestFailover) 1 jobs: ['6494217'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/failover}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on gibba017 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:f03f788619bd071f853863160f6e14ae60dc1efc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d00f0282-4199-11ec-8c2c-001a4aab830c -- ceph orch daemon add osd gibba017:/dev/nvme0n1p4' 1 jobs: ['6494210'] suites: ['1-mount', '2-workload/dbench', 'distro/centos_latest}', 'fs/cephadm/multivolume/{0-start'] Crash: "2021-11-09T20:48:35.873591+0000 mon.a (mon.0) 1278 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log ceph version 16.2.6-468-gf03f7886 (f03f788619bd071f853863160f6e14ae60dc1efc) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f25e1b4b3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x7f25e1dbf564] 5: (Client::_do_remount(bool)+0x518) [0x555568885968] 6: (Context::complete(int)+0xd) [0x55556890755d] 7: (Finisher::finisher_thread_entry()+0x19d) [0x7f25e1e7274d] 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f25e1b3f609] 9: clone() 1 jobs: ['6494216'] suites: ['clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/cephfs_misc_tests}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Command failed on gibba027 with status 32: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,name=1,conf=/etc/ceph/ceph.conf,norbytes,fsc=client1,nowsync' 1 jobs: ['6494215'] suites: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'whitelist_health', 'whitelist_wrongly_marked_down}']