Found 232 jobs Found 3 distinct failure reasons Failure: Command failed on smithi023 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 4 jobs: ['5471319', '5471390', '5471474', '5471303'] suites intersection: ['conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['ceph-thrash/mon', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-4-client-coloc', 'clusters/fixed-2-ucephfs', 'conf/{client', 'fs/basic_functional/{begin', 'fs/basic_workload/{begin', 'fs/thrash/{begin', 'inline/no', 'inline/yes', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'no_client_pidfile', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/filestore-xfs', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'omap_limit/10000', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/cfuse_workunit_suites_fsync}', 'tasks/cfuse_workunit_suites_iogen}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/openfiletable}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_subvolume_retain_snapshot_invalid_recreate (tasks.cephfs.test_volumes.TestVolumes) 1 jobs: ['5471370'] suites: ['clusters/1-mds-4-client-coloc', 'conf/{client', 'fs/basic_functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{centos_8}', 'tasks/volumes}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi110 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=53f857cc0602dd77284fd703c292252370ced0a5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' ceph version 16.0.0-5932-g53f857cc (53f857cc0602dd77284fd703c292252370ced0a5) pacific (dev) 1: () 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x7f125b204db9] 5: () 6: () 7: () 8: (DispatchQueue::entry()+0x126a) [0x7f125b42db1a] 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f125b4dcd31] 10: () 11: clone() 1 jobs: ['5471511'] suites: ['ceph-thrash/mon', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'fs/thrash/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'supported-random-distros$/{rhel_8}', 'tasks/cfuse_workunit_snaptests}', 'whitelist_health', 'whitelist_wrongly_marked_down}']