2023-10-09T05:39:46.877 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-09T05:39:46.877 INFO:teuthology.results:subset: '111/12000' 2023-10-09T05:39:46.877 INFO:teuthology.results:seed: '8174' 2023-10-09T06:26:57.529 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-09T06:26:57.558 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-09T08:19:05.904 INFO:teuthology.results:Tests finished! gathering results... 2023-10-09T08:19:07.427 INFO:scrape:Found 19 jobs 2023-10-09T08:19:18.597 INFO:scrape:Found 9 distinct failure reasons 2023-10-09T08:19:18.597 INFO:scrape:Failure: "1696832708.651863 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-10-09T08:19:18.597 INFO:scrape:2 jobs: ['7418369', '7418378'] 2023-10-09T08:19:18.598 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'objectstore/bluestore-bitmap'] 2023-10-09T08:19:18.598 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{rhel_8}', 'tasks/mirror}', 'tasks/snap-schedule}'] 2023-10-09T08:19:18.598 INFO:scrape: 2023-10-09T08:19:18.598 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=08cac0187919fc3a247efdf445f652e2fb30a647 2023-10-09T08:19:18.598 INFO:scrape:2 jobs: ['7418374', '7418381'] 2023-10-09T08:19:18.598 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-10-09T08:19:18.598 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-10-09T08:19:18.599 INFO:scrape: 2023-10-09T08:19:18.599 INFO:scrape:Failure: Command failed on smithi110 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-10-09T08:19:18.599 INFO:scrape:2 jobs: ['7418373', '7418382'] 2023-10-09T08:19:18.599 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-10-09T08:19:18.599 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}'] 2023-10-09T08:19:18.599 INFO:scrape: 2023-10-09T08:19:18.599 INFO:scrape:Failure: Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes) 2023-10-09T08:19:18.599 INFO:scrape:1 jobs: ['7418377'] 2023-10-09T08:19:18.600 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-10-09T08:19:18.600 INFO:scrape: 2023-10-09T08:19:18.600 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-10-09T08:19:18.600 INFO:scrape:1 jobs: ['7418380'] 2023-10-09T08:19:18.600 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-10-09T08:19:18.600 INFO:scrape: 2023-10-09T08:19:18.600 INFO:scrape:Crash: Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62a27f0d8453e2104f759ed08ba781e24c5ed056 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' 2023-10-09T08:19:18.600 INFO:scrape:ceph version 17.2.6-1320-g08cac018 (08cac0187919fc3a247efdf445f652e2fb30a647) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55e471c435ad] 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x9fb) [0x55e472342fab] 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2df) [0x55e47235414f] 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55e471f0e5c3] 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7c2) [0x55e472165292] 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x50d) [0x55e471e861dd] 7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xd25) [0x55e471ee3405] 8: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x288d) [0x55e471eeaa6d] 9: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c0) [0x55e471d46080] 10: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x6d) [0x55e47204e9ed] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x112f) [0x55e471d80d5f] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55e4724c4315] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55e4724c6a34] 14: /lib64/libpthread.so.0(+0x81ca) [0x7f54126081ca] 2023-10-09T08:19:18.601 INFO:scrape:1 jobs: ['7418367'] 2023-10-09T08:19:18.601 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{centos_8}', 'fs/full/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] 2023-10-09T08:19:18.601 INFO:scrape: 2023-10-09T08:19:18.601 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-10-09T08:19:18.601 INFO:scrape:1 jobs: ['7418372'] 2023-10-09T08:19:18.601 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2023-10-09T08:19:18.601 INFO:scrape: 2023-10-09T08:19:18.601 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi084 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62a27f0d8453e2104f759ed08ba781e24c5ed056 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-10-09T08:19:18.602 INFO:scrape:1 jobs: ['7418379'] 2023-10-09T08:19:18.602 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-10-09T08:19:18.602 INFO:scrape: 2023-10-09T08:19:18.602 INFO:scrape:Failure: "1696832441.3288944 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 222697.888416 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-10-09T08:19:18.602 INFO:scrape:1 jobs: ['7418375'] 2023-10-09T08:19:18.602 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] 2023-10-09T08:19:18.602 INFO:scrape: 2023-10-09T08:19:18.603 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/ failed: 12 dead: 0 running: 0 waiting: 0 queued: 0 passed: 7 Fail ================================================================= [7418367] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:35:26 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418367/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418367/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b8b9f093738409683eedc074ca7cd65 Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62a27f0d8453e2104f759ed08ba781e24c5ed056 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' [7418369] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{rhel_8} tasks/mirror} ----------------------------------------------------------------- time: 02:09:35 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418369/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418369/ "1696832708.651863 mon.a (mon.0) 1978 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7418372] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:27:46 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418372/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418372/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff9294f6fa1c451fa83c45c911a3cf25 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7418373] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:17 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418373/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418373/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c10d72b5c2a942b0ac89657e119955a0 Command failed on smithi110 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7418374] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:43 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418374/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418374/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28506b47bacb4ae88927ad7d6d50639e Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =08cac0187919fc3a247efdf445f652e2fb30a647 [7418375] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:23:08 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418375/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418375/ "1696832441.3288944 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 222697.888416 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7418377] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:23:25 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418377/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418377/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e68320d3c7b5416b846fa78dd8c23c89 Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes) [7418378] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 01:50:43 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418378/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418378/ "1696838385.8347495 mon.a (mon.0) 1382 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7418379] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:20:21 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418379/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418379/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6078343f684c484896de58d9f7a6db7b Command failed (workunit test libcephfs/test.sh) on smithi084 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62a27f0d8453e2104f759ed08ba781e24c5ed056 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7418380] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:34:04 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418380/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418380/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d29043efe0446bd92fa37116333a59b Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) [7418381] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:44 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418381/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418381/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6d2c3b0fc0a442f89764f3914f0ccc0f Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =08cac0187919fc3a247efdf445f652e2fb30a647 [7418382] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:42 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418382/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418382/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da90aa6cffe04bdc8583832d46b737d1 Command failed on smithi072 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' Pass ================================================================= [7418364] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:43:32 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418364/ [7418365] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 00:56:33 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418365/ [7418366] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:31:32 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418366/ [7418368] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:32:42 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418368/ [7418370] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} time: 00:28:33 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418370/ [7418371] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:17 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418371/ [7418376] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:45:23 info: http://pulpito.ceph.com/vshankar-2023-10-09_05:36:44-fs-wip-yuri-testing-2023-10-06-0949-quincy-testing-default-smithi/7418376/