Found 38 jobs Found 7 distinct failure reasons Failure: "2022-06-07T12:28:47.584549+0000 mds.f (mds.0) 14 : cluster [WRN] client.4711 isn't responding to mclientcaps(revoke), ino 0x10000004d4e pending pAsLsXsFsc issued pAsLsXsFscb, sent 300.162164 seconds ago" in cluster log 1 jobs: ['6865751'] suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] Failure: "2022-06-07T11:46:40.681235+0000 mon.a (mon.0) 2410 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log 1 jobs: ['6865738'] suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}'] Failure: "2022-06-07T11:32:36.434735+0000 osd.4 (osd.4) 22 : cluster [ERR] 5.14s0 deep-scrub : stat mismatch, got 0/1 objects, 0/0 clones, 0/1 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/4194304 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 1 jobs: ['6865728'] suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2 jobs: ['6865745', '6865734'] suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag'] suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'ranks/3', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'workunit/kernel_untar_build}}'] Failure: Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring) 1 jobs: ['6865720'] suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] Crash: "2022-06-07T12:54:47.382888+0000 mon.a (mon.0) 4073 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log ceph version 17.0.0-12847-gaf21e3f3 (af21e3f3c94a19a2de88f6f97fc2f35a9d22c9c2) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f7062fdec20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f706403bb72] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2c1d35) [0x7f706403bd35] 6: ceph-mds(+0x28f3b3) [0x5622ba7f53b3] 7: (MDCache::dispatch_fragment_dir(boost::intrusive_ptr&)+0x4f9) [0x5622ba883fc9] 8: (MDCache::fragment_frozen(boost::intrusive_ptr&, int)+0x2d4) [0x5622ba884e04] 9: (MDCache::fragment_mark_and_complete(boost::intrusive_ptr&)+0xa5c) [0x5622ba89b0ec] 10: (MDCache::merge_dir(CInode*, frag_t)+0x5ab) [0x5622ba89bf2b] 11: ceph-mds(+0x41b272) [0x5622ba981272] 12: (Context::complete(int)+0xd) [0x5622ba73357d] 13: (CommonSafeTimer::timer_thread()+0x18b) [0x7f706415367b] 14: (CommonSafeTimerThread::entry()+0x11) [0x7f7064155681] 15: /lib64/libpthread.so.0(+0x817a) [0x7f7062fd417a] 16: clone() 1 jobs: ['6865742'] suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] Dead: b'2022-06-07T17:34:28.210 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['6865727'] suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/3', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/kernel_untar_build}}', 'wsync/no}']