Found 171 jobs Missing teuthology log /home/teuthworker/archive/yuriw-2023-08-10_20:15:49-fs-wip-yuri2-testing-2023-08-08-0755-pacific-distro-default-smithi/7365886/teuthology.log Found 7 distinct failure reasons Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 1 jobs: ['7365884'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}'] Crash: Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi057 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=81b69661cc883eb16f4c3a43b0bca7ed029d0a11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' ceph version 16.2.13-455-g81b69661 (81b69661cc883eb16f4c3a43b0bca7ed029d0a11) pacific (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f3403eb7cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x558b2a6c362b] 5: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x1497) [0x558b2ad14247] 6: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x316) [0x558b2ad314d6] 7: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x58) [0x558b2a972568] 8: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0xcad) [0x558b2ab62fbd] 9: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xcf0) [0x558b2a8dda90] 10: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x115d) [0x558b2a93a04d] 11: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x2dd3) [0x558b2a942bb3] 12: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xd1c) [0x558b2a949efc] 13: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x309) [0x558b2a7cd7a9] 14: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x6b) [0x558b2aa2fedb] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x558b2a7eaf68] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x558b2ae6ff34] 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x558b2ae72e14] 18: /lib64/libpthread.so.0(+0x81ca) [0x7f3403ead1ca] 19: clone() 1 jobs: ['7365825'] suites: ['clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{centos_8}', 'fs/full/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] Failure: Command failed (workunit test libcephfs/test.sh) on smithi132 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=81b69661cc883eb16f4c3a43b0bca7ed029d0a11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 1 jobs: ['7365656'] suites: ['clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] Crash: "2023-08-11T01:47:31.920484+0000 mds.c (mds.0) 1 : cluster [ERR] MDS abort because newly corrupt dentry to be committed: [dentry #0x1/a [fffffffffffffff6,head] auth (dversion lock) v=13 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5634d6c01680]" in cluster log ceph version 16.2.13-455-g81b69661 (81b69661cc883eb16f4c3a43b0bca7ed029d0a11) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f2381e2a420] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1af) [0x7f23820dc584] 5: (CDentry::check_corruption(bool)+0x86b) [0x5634d4e5607b] 6: ceph-mds(+0x48ad7a) [0x5634d4e79d7a] 7: (CDir::_omap_commit(int)+0x318) [0x5634d4e7a4b8] 8: (CDir::_commit(unsigned long, int)+0x289) [0x5634d4e7bb09] 9: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x138) [0x5634d4e7c188] 10: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x170f) [0x5634d4fc34ff] 11: (MDLog::try_expire(LogSegment*, int)+0x6e) [0x5634d4f4a05e] 12: (MDLog::trim_all()+0xa18) [0x5634d4f4edf8] 13: (LambdaContext::finish(int)+0x231) [0x5634d4c03c01] 14: (Context::complete(int)+0xd) [0x5634d4bb9cdd] 15: (MDSContext::complete(int)+0x61) [0x5634d4f2e2c1] 16: (C_IO_Wrapper::finish(int)+0x12) [0x5634d4f2f022] 17: (MDSContext::complete(int)+0x61) [0x5634d4f2e2c1] 18: (MDSIOContextBase::complete(int)+0x4fc) [0x5634d4f2e9ec] 19: (C_IO_Wrapper::complete(int)+0x105) [0x5634d4f2efa5] 20: (Finisher::finisher_thread_entry()+0x19d) [0x7f238219099d] 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f2381e1e609] 22: clone() 1 jobs: ['7365654'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/damage}'] Dead: None 1 jobs: ['7365886'] Failure: "2023-08-11T01:49:43.585060+0000 mon.a (mon.0) 480 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 2 jobs: ['7365652', '7365800'] suites intersection: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health'] suites union: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/client-recovery}', 'tasks/snap-schedule}'] Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 1 jobs: ['7365852'] suites: ['centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']