2023-12-24T17:51:49.757 INFO:root:teuthology version: 0.0.1.dev226+g9680d2a 2023-12-24T17:51:49.758 INFO:teuthology.results:subset: '908/1024' 2023-12-24T17:51:49.758 INFO:teuthology.results:seed: '9956' 2023-12-24T20:06:10.009 INFO:root:teuthology version: 0.0.1.dev226+g9680d2a 2023-12-24T20:06:10.033 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-12-25T02:11:34.985 INFO:teuthology.results:Tests finished! gathering results... 2023-12-25T02:11:44.672 INFO:scrape:Found 104 jobs 2023-12-25T02:11:57.419 INFO:scrape:Found 9 distinct failure reasons 2023-12-25T02:11:57.419 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2023-12-25T02:11:57.419 INFO:scrape:1 jobs: ['7500424'] 2023-12-25T02:11:57.419 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2023-12-25T02:11:57.419 INFO:scrape: 2023-12-25T02:11:57.420 INFO:scrape:Failure: "1703448996.3660026 mon.a (mon.0) 744 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-12-25T02:11:57.420 INFO:scrape:3 jobs: ['7500466', '7500302', '7500316'] 2023-12-25T02:11:57.420 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-12-25T02:11:57.420 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{rhel_8}', 'tasks/metrics}', 'tasks/mirror}', 'tasks/snap-schedule}'] 2023-12-25T02:11:57.420 INFO:scrape: 2023-12-25T02:11:57.420 INFO:scrape:Failure: Command failed on smithi160 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-12-25T02:11:57.420 INFO:scrape:2 jobs: ['7500446', '7500341'] 2023-12-25T02:11:57.420 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-12-25T02:11:57.420 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-12-25T02:11:57.420 INFO:scrape: 2023-12-25T02:11:57.420 INFO:scrape:Failure: Test failure: test_mirroring_init_failure (tasks.cephfs.test_mirroring.TestMirroring) 2023-12-25T02:11:57.421 INFO:scrape:1 jobs: ['7500306'] 2023-12-25T02:11:57.421 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2023-12-25T02:11:57.421 INFO:scrape: 2023-12-25T02:11:57.421 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-12-25T02:11:57.421 INFO:scrape:1 jobs: ['7500440'] 2023-12-25T02:11:57.421 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'ranks/3', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-12-25T02:11:57.421 INFO:scrape: 2023-12-25T02:11:57.421 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds 2023-12-25T02:11:57.421 INFO:scrape:1 jobs: ['7500410'] 2023-12-25T02:11:57.421 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/fs/misc}', 'wsync/{no}}'] 2023-12-25T02:11:57.421 INFO:scrape: 2023-12-25T02:11:57.421 INFO:scrape:Failure: "1703447813.2546344 osd.1 (osd.1) 3 : cluster [WRN] OSD bench result of 215993.357772 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.1. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-12-25T02:11:57.422 INFO:scrape:1 jobs: ['7500456'] 2023-12-25T02:11:57.422 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-12-25T02:11:57.422 INFO:scrape: 2023-12-25T02:11:57.422 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-12-25T02:11:57.422 INFO:scrape:1 jobs: ['7500337'] 2023-12-25T02:11:57.422 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-12-25T02:11:57.422 INFO:scrape: 2023-12-25T02:11:57.422 INFO:scrape:Crash: Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' 2023-12-25T02:11:57.422 INFO:scrape:ceph version 17.2.7-168-g5e7ce129 (5e7ce1294a22dbbbddee64f3a18ddd2050b07f8f) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5587c0e7befb] 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x9fb) [0x5587c157bfdb] 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2df) [0x5587c158d17f] 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x5587c1147203] 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7c2) [0x5587c139e282] 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x50d) [0x5587c10bedad] 7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xd25) [0x5587c111c045] 8: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x288d) [0x5587c11236ad] 9: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c0) [0x5587c0f7eb30] 10: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x6d) [0x5587c12879ad] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x112f) [0x5587c0fb98ff] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5587c16fd995] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5587c17000b4] 14: /lib64/libpthread.so.0(+0x81ca) [0x7fab468981ca] 2023-12-25T02:11:57.422 INFO:scrape:1 jobs: ['7500300'] 2023-12-25T02:11:57.422 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{centos_8}', 'fs/full/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] 2023-12-25T02:11:57.422 INFO:scrape: