Found 173 jobs Found 9 distinct failure reasons Failure: "2023-11-15T03:30:42.388875+0000 mon.a (mon.0) 1534 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log 5 jobs: ['7458193', '7458075', '7458100', '7458145', '7458041'] suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health'] suites union: ['clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin', 'fs/shell/{begin', 'fs/volumes/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/cephfs-shell}', 'tasks/metrics}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'test/clone}}'] Failure: Command failed on smithi032 with status 5: 'sudo systemctl stop ceph-6e3dab36-835e-11ee-8dbe-212e2dc638e7@mon.smithi032' 2 jobs: ['7458113', '7458065'] suites intersection: ['1-volume/{0-create', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4'] suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4'] Failure: Command failed on smithi069 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2 jobs: ['7458040', '7458129'] suites intersection: ['conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] suites union: ['clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] Crash: "2023-11-15T03:00:42.050657+0000 mon.a (mon.0) 686 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log ceph version 16.2.14-170-g620f25c1 (620f25c1794f62207424fc9319427943fb8d1fae) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x7f13680684b5] 2: (CDentry::check_corruption(bool)+0x86b) [0x56060b9a0cbb] 3: ceph-mds(+0x48daaa) [0x56060b9c4aaa] 4: (CDir::_omap_commit(int)+0x318) [0x56060b9c51e8] 5: (CDir::_commit(unsigned long, int)+0x289) [0x56060b9c6839] 6: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x138) [0x56060b9c6eb8] 7: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x170f) [0x56060bb0ff9f] 8: (MDLog::try_expire(LogSegment*, int)+0x6e) [0x56060ba9901e] 9: (MDLog::trim_all()+0xa18) [0x56060ba9af48] 10: (LambdaContext::finish(int)+0x231) [0x56060b74d3c1] 11: (Context::complete(int)+0xd) [0x56060b70317d] 12: (MDSContext::complete(int)+0x61) [0x56060ba7a4b1] 13: (C_IO_Wrapper::finish(int)+0x12) [0x56060ba7b212] 14: (MDSContext::complete(int)+0x61) [0x56060ba7a4b1] 15: (MDSIOContextBase::complete(int)+0x4fc) [0x56060ba7abdc] 16: (C_IO_Wrapper::complete(int)+0x105) [0x56060ba7b195] 17: (Finisher::finisher_thread_entry()+0x19d) [0x7f136811ca9d] 18: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f1370bc1609] 1 jobs: ['7458153'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{ignorelist_health', 'tasks/damage}'] Failure: smithi177.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation releasever: null security: false skip_broken: false state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''baseos'': Yum repo downloading error: Downloading error(s): repodata/f4fc5bc99f714c7a5624760fe4edd64b7389db181e1faa3c85fd33f59f2f9f9e-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/22c687befd5b6db5b468a434d35e680e70eee0cf4d66008ffee7b27bc5c03ebc-filelists.xml.gz - Cannot download, all mirrors were already tried without success' rc: 1 results: [] 1 jobs: ['7458141'] suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] Failure: "2023-11-15T02:26:33.164780+0000 mds.f (mds.0) 4 : cluster [WRN] client.4764 isn't responding to mclientcaps(revoke), ino 0x1000000f56d pending pAsLsXsFscr issued pAsLsXsFscrb, sent 302.709171 seconds ago" in cluster log 1 jobs: ['7458071'] suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}'] Failure: timeout expired in wait_until_healthy 1 jobs: ['7458082'] suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'roles', 'tasks/{0-from/v16.2.4'] Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 1 jobs: ['7458114'] suites: ['clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/acls}'] Crash: Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi148 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=620f25c1794f62207424fc9319427943fb8d1fae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' ceph version 16.2.14-170-g620f25c1 (620f25c1794f62207424fc9319427943fb8d1fae) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe0) [0x5557209210f5] 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x95a) [0x555720ff8aaa] 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2fd) [0x555720ffb1dd] 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x58) [0x555720bd7408] 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7e9) [0x555720dfa479] 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xecf) [0x555720b2f7ff] 7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xf95) [0x555720b96df5] 8: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x36b9) [0x555720b9b279] 9: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xe61) [0x555720ba62c1] 10: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c0) [0x5557209f5c70] 11: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x66) [0x555720cadf46] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8aa) [0x555720a232ca] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x403) [0x55572115e653] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x555721161694] 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f5d90ace609] 1 jobs: ['7458092'] suites: ['clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/full/{begin', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}']