Found 30 jobs Found 11 distinct failure reasons Failure: "2021-06-11T04:17:39.442703+0000 mgr.x (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: rook" in cluster log 6 jobs: ['6165769', '6165775', '6165779', '6165784', '6165764', '6165759'] suites intersection: ['1-client', '2-upgrade', 'clusters/1-mds-2-client-micro', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/pacific}}', '4-compat_client', '5-client-sanity}}', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{frag_enable', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) 1 jobs: ['6165776'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_scrub_backtrace (tasks.cephfs.test_multimds_misc.TestScrub2) 1 jobs: ['6165773'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/multimds_misc}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_r_with_fsname_and_no_path_in_cap (tasks.cephfs.test_multifs_auth.TestMDSCaps) 3 jobs: ['6165763', '6165783', '6165774'] suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'mds', 'mon', 'mon-debug', 'osd}', 'overrides/{frag_enable', 'tasks/multifs-auth}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'rhel_8}', 'tasks/multifs-auth}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 1 jobs: ['6165777'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) 2 jobs: ['6165756', '6165760'] suites intersection: ['mount/fuse'] suites union: ['centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'debug', 'fs/mirror/{begin', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/whitelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}', 'tasks/mirror}}'] Crash: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan), test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) ceph version 17.0.0-5093-gd0a0802e (d0a0802e989a68354da1267cad401268aada61fe) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f6f866c93c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f6f86976d48] 5: /usr/lib/ceph/libceph-common.so.2(+0x264ee6) [0x7f6f86976ee6] 6: (MDCache::predirty_journal_parents(boost::intrusive_ptr, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0x2209) [0x56095ec45c09] 7: (Server::_unlink_local(boost::intrusive_ptr&, CDentry*, CDentry*)+0x9c2) [0x56095eba0ce2] 8: (Server::handle_client_unlink(boost::intrusive_ptr&)+0x414) [0x56095ebbfa74] 9: (Server::dispatch_client_request(boost::intrusive_ptr&)+0x9b0) [0x56095ebdc200] 10: (Server::handle_client_request(boost::intrusive_ptr const&)+0x5dd) [0x56095ebdcfad] 11: (Server::dispatch(boost::intrusive_ptr const&)+0xec) [0x56095ebe5b8c] 12: (MDSRank::handle_message(boost::intrusive_ptr const&)+0x3e9) [0x56095eb32089] 13: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x643) [0x56095eb34e03] 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56095eb3549c] 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x128) [0x56095eb25ce8] 16: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x460) [0x7f6f86bc4ff0] 17: (DispatchQueue::entry()+0x58f) [0x7f6f86bc288f] 18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f6f86c8d7e1] 19: (Thread::_entry_func(void*)+0xd) [0x7f6f86a5fd8d] 20: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f6f866bd609] 21: clone() 2 jobs: ['6165766', '6165780'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/data-scan}', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_deauthorize_auth_id_after_out_of_band_update (tasks.cephfs.test_volumes.TestSubvolumes) 2 jobs: ['6165761', '6165781'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/volumes/{begin', 'mds', 'mon', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/volumes/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_clients (tasks.cephfs.test_failover.TestMultiFilesystems) 3 jobs: ['6165778', '6165768', '6165757'] suites intersection: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin', 'mds', 'mon', 'mon-debug', 'osd}', 'overrides/{frag_enable', 'tasks/failover}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] suites union: ['clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin', 'k-testing}', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'tasks/failover}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 1 jobs: ['6165767'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/forward-scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] Failure: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull) 1 jobs: ['6165771'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'tasks/mds-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}']