2023-10-08T15:42:54.766 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-08T15:42:54.766 INFO:teuthology.results:subset: '111/12000' 2023-10-08T15:42:54.766 INFO:teuthology.results:seed: '8174' 2023-10-08T15:43:43.065 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-08T15:43:43.069 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-08T18:30:56.100 INFO:teuthology.results:Tests finished! gathering results... 2023-10-08T18:30:57.618 INFO:scrape:Found 19 jobs 2023-10-08T18:31:11.628 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-10-08_15:41:30-fs-wip-yuri-testing-2023-10-06-0949-quincy-distro-default-smithi/7418297/teuthology.log 2023-10-08T18:31:11.629 INFO:scrape:Found 10 distinct failure reasons 2023-10-08T18:31:11.629 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi133 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=abf19fc7a04fe4fceab4e98d4013ae1da7c06006 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-10-08T18:31:11.629 INFO:scrape:1 jobs: ['7418300'] 2023-10-08T18:31:11.629 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-10-08T18:31:11.629 INFO:scrape: 2023-10-08T18:31:11.630 INFO:scrape:Failure: Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes) 2023-10-08T18:31:11.630 INFO:scrape:1 jobs: ['7418298'] 2023-10-08T18:31:11.630 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-10-08T18:31:11.630 INFO:scrape: 2023-10-08T18:31:11.630 INFO:scrape:Failure: "1696781087.9182277 osd.1 (osd.1) 3 : cluster [WRN] OSD bench result of 222829.556558 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.1. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-10-08T18:31:11.630 INFO:scrape:1 jobs: ['7418296'] 2023-10-08T18:31:11.630 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] 2023-10-08T18:31:11.630 INFO:scrape: 2023-10-08T18:31:11.631 INFO:scrape:Failure: Command failed on smithi092 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-10-08T18:31:11.631 INFO:scrape:2 jobs: ['7418294', '7418303'] 2023-10-08T18:31:11.631 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-10-08T18:31:11.631 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-10-08T18:31:11.631 INFO:scrape: 2023-10-08T18:31:11.631 INFO:scrape:Failure: Test failure: test_client_blocklisted_oldest_tid (tasks.cephfs.test_client_limits.TestClientLimits) 2023-10-08T18:31:11.632 INFO:scrape:1 jobs: ['7418289'] 2023-10-08T18:31:11.632 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2023-10-08T18:31:11.632 INFO:scrape: 2023-10-08T18:31:11.632 INFO:scrape:Failure: "1696784213.6405792 mon.a (mon.0) 1975 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-10-08T18:31:11.632 INFO:scrape:3 jobs: ['7418293', '7418290', '7418299'] 2023-10-08T18:31:11.632 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-10-08T18:31:11.632 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'debug', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/ignorelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{rhel_8}', 'tasks/mirror}', 'tasks/mirror}}', 'tasks/snap-schedule}'] 2023-10-08T18:31:11.632 INFO:scrape: 2023-10-08T18:31:11.632 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=08cac0187919fc3a247efdf445f652e2fb30a647 2023-10-08T18:31:11.633 INFO:scrape:2 jobs: ['7418295', '7418302'] 2023-10-08T18:31:11.633 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-10-08T18:31:11.633 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-10-08T18:31:11.633 INFO:scrape: 2023-10-08T18:31:11.633 INFO:scrape:Crash: Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=abf19fc7a04fe4fceab4e98d4013ae1da7c06006 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' 2023-10-08T18:31:11.633 INFO:scrape:ceph version 17.2.6-1320-g08cac018 (08cac0187919fc3a247efdf445f652e2fb30a647) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x555b2209b5ad] 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x9fb) [0x555b2279afab] 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2df) [0x555b227ac14f] 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x555b223665c3] 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7c2) [0x555b225bd292] 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x50d) [0x555b222de1dd] 7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xd25) [0x555b2233b405] 8: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x288d) [0x555b22342a6d] 9: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c0) [0x555b2219e080] 10: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x6d) [0x555b224a69ed] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x112f) [0x555b221d8d5f] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x555b2291c315] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x555b2291ea34] 14: /lib64/libpthread.so.0(+0x81ca) [0x7f6d8bbcc1ca] 2023-10-08T18:31:11.633 INFO:scrape:1 jobs: ['7418288'] 2023-10-08T18:31:11.633 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{centos_8}', 'fs/full/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] 2023-10-08T18:31:11.634 INFO:scrape: 2023-10-08T18:31:11.634 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-10-08T18:31:11.634 INFO:scrape:1 jobs: ['7418301'] 2023-10-08T18:31:11.634 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-10-08T18:31:11.634 INFO:scrape: 2023-10-08T18:31:11.634 INFO:scrape:Dead: None 2023-10-08T18:31:11.634 INFO:scrape:1 jobs: ['7418297'] 2023-10-08T18:31:11.634 INFO:scrape: