Found 171 jobs Missing teuthology log /home/teuthworker/archive/yuriw-2023-07-20_14:36:46-fs-wip-yuri-testing-2023-07-19-1340-pacific-distro-default-smithi/7344933/teuthology.log Found 9 distinct failure reasons Dead: None 7 jobs: ['7344933', '7344935', '7344924', '7344865', '7344871', '7344880', '7344919'] Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 1 jobs: ['7344895'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}'] Failure: {"Creating volume group 'vg_nvme' failed", "Run `vgcreate --help' for more information.", '/dev/vg_nvme: already exists in filesystem'} 1 jobs: ['7344934'] suites: ['2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag', 'prefetch_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] Crash: Test failure: test_rebuild_simple (tasks.cephfs.test_data_scan.TestDataScan) ceph version 16.2.13-374-g5b6522b2 (5b6522b2ea39bacb47ee81e83150dff111806203) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7f8c24ca9b20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f8c25cbb715] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2798de) [0x7f8c25cbb8de] 6: (MDLog::_recovery_thread(MDSContext*)+0xd72) [0x5574062473e2] 7: (MDLog::RecoveryThread::entry()+0x15) [0x557405f390b5] 8: /lib64/libpthread.so.0(+0x814a) [0x7f8c24c9f14a] 9: clone() 1 jobs: ['7344792'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/data-scan}'] Failure: "2023-07-20T17:05:15.097718+0000 mon.a (mon.0) 312 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2 jobs: ['7344855', '7344796'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'test/clone}}'] Failure: {'Failed to update apt cache: unknown reason'} 1 jobs: ['7344872'] suites: ['clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] Failure: timed out 1 jobs: ['7344922'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/cap-flush}'] Failure: "2023-07-20T17:02:46.065611+0000 mds.e (mds.0) 1 : cluster [WRN] client.4752 isn't responding to mclientcaps(revoke), ino 0x1000000f41c pending pAsLsXsFsc issued pAsLsXsFscb, sent 304.945746 seconds ago" in cluster log 1 jobs: ['7344795'] suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] Crash: "2023-07-20T15:43:34.770772+0000 mds.a (mds.0) 1 : cluster [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x55ea80f0b400]" in cluster log ceph version 16.2.13-374-g5b6522b2 (5b6522b2ea39bacb47ee81e83150dff111806203) pacific (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fc1ef737cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x7fc1f074eed5] 5: (CDentry::check_corruption(bool)+0x456) [0x55ea7eddcab6] 6: ceph-mds(+0x433722) [0x55ea7edf8722] 7: (CDir::_omap_commit(int)+0x6b7) [0x55ea7edf9117] 8: (CDir::_commit(unsigned long, int)+0x595) [0x55ea7edf9c35] 9: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x130) [0x55ea7edfa110] 10: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x1ba2) [0x55ea7ef2a882] 11: (MDLog::try_expire(LogSegment*, int)+0x6d) [0x55ea7eebba4d] 12: (MDLog::trim_all()+0x816) [0x55ea7eebd6a6] 13: (LambdaContext::finish(int)+0x195) [0x55ea7ebc9415] 14: (Context::complete(int)+0xd) [0x55ea7eb8967d] 15: (MDSContext::complete(int)+0x203) [0x55ea7eea41c3] 16: (C_IO_Wrapper::finish(int)+0x12) [0x55ea7eea4f52] 17: (MDSContext::complete(int)+0x203) [0x55ea7eea41c3] 18: (MDSIOContextBase::complete(int)+0x6ac) [0x55ea7eea496c] 19: (C_IO_Wrapper::complete(int)+0x12d) [0x55ea7eea4e9d] 20: (Finisher::finisher_thread_entry()+0x1a5) [0x7fc1f07f0635] 21: /lib64/libpthread.so.0(+0x81ca) [0x7fc1ef72d1ca] 22: clone() 1 jobs: ['7344784'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/damage}']