2023-10-14T16:53:47.561 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-14T16:53:47.563 INFO:teuthology.results:subset: '111/12000' 2023-10-14T16:53:47.563 INFO:teuthology.results:seed: '2462' 2023-10-14T19:43:45.061 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-14T19:43:45.086 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-15T07:43:47.012 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-10-15T07:44:06.536 INFO:scrape:Found 189 jobs 2023-10-15T07:44:13.607 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425835/teuthology.log 2023-10-15T07:44:24.166 INFO:scrape:Found 13 distinct failure reasons 2023-10-15T07:44:24.167 INFO:scrape:Failure: Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) 2023-10-15T07:44:24.168 INFO:scrape:1 jobs: ['7425840'] 2023-10-15T07:44:24.168 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/strays}'] 2023-10-15T07:44:24.169 INFO:scrape: 2023-10-15T07:44:24.169 INFO:scrape:Failure: "1697307455.6124878 mon.a (mon.0) 2225 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2023-10-15T07:44:24.169 INFO:scrape:6 jobs: ['7425735', '7425752', '7425778', '7425823', '7425746', '7425824'] 2023-10-15T07:44:24.170 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-10-15T07:44:24.170 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'debug', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/ignorelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{ignorelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/client-recovery}', 'tasks/journal-repair}', 'tasks/mirror}', 'tasks/mirror}}', 'tasks/snap-schedule}', 'tasks/volumes/{overrides', 'test/clone}}'] 2023-10-15T07:44:24.171 INFO:scrape: 2023-10-15T07:44:24.171 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 2023-10-15T07:44:24.172 INFO:scrape:2 jobs: ['7425843', '7425775'] 2023-10-15T07:44:24.172 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-10-15T07:44:24.172 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-10-15T07:44:24.173 INFO:scrape: 2023-10-15T07:44:24.173 INFO:scrape:Failure: Command failed on smithi088 with status 5: 'sudo systemctl stop ceph-17bdafae-6ac2-11ee-8db6-212e2dc638e7@mon.b' 2023-10-15T07:44:24.174 INFO:scrape:4 jobs: ['7425784', '7425836', '7425770', '7425861'] 2023-10-15T07:44:24.174 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}'] 2023-10-15T07:44:24.174 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-client', '2-logrotate}', '2-upgrade', '3-compat_client/no}}', 'begin/{0-install', 'centos_latest', 'clusters/1-mds-2-client', 'clusters/1-mds-2-client-micro', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'multimds/no', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/ior-shared-file}', 'tasks/{0-check-counter', 'tasks/{0-octopus', 'workunit/kernel_untar_build}', 'wsync/{yes}}'] 2023-10-15T07:44:24.175 INFO:scrape: 2023-10-15T07:44:24.175 INFO:scrape:Dead: b'2023-10-15T07:40:53.622 DEBUG:teuthology.exit:Finished running handlers' 2023-10-15T07:44:24.176 INFO:scrape:1 jobs: ['7425857'] 2023-10-15T07:44:24.176 INFO:scrape:suites: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/fsx}', 'wsync/{yes}}'] 2023-10-15T07:44:24.177 INFO:scrape: 2023-10-15T07:44:24.177 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi045 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-10-15T07:44:24.177 INFO:scrape:1 jobs: ['7425830'] 2023-10-15T07:44:24.178 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-10-15T07:44:24.178 INFO:scrape: 2023-10-15T07:44:24.179 INFO:scrape:Failure: SSH connection to smithi163 was lost: 'sudo chmod 0666 /tmp/tmp.KWm4mPiufy' 2023-10-15T07:44:24.179 INFO:scrape:1 jobs: ['7425834'] 2023-10-15T07:44:24.179 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-10-15T07:44:24.180 INFO:scrape: 2023-10-15T07:44:24.180 INFO:scrape:Failure: "1697309097.1160598 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 218088.856815 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-10-15T07:44:24.181 INFO:scrape:1 jobs: ['7425783'] 2023-10-15T07:44:24.181 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] 2023-10-15T07:44:24.181 INFO:scrape: 2023-10-15T07:44:24.182 INFO:scrape:Failure: Command failed on smithi139 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7967dbbc-6abd-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json' 2023-10-15T07:44:24.182 INFO:scrape:1 jobs: ['7425747'] 2023-10-15T07:44:24.183 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific'] 2023-10-15T07:44:24.183 INFO:scrape: 2023-10-15T07:44:24.183 INFO:scrape:Dead: None 2023-10-15T07:44:24.184 INFO:scrape:1 jobs: ['7425835'] 2023-10-15T07:44:24.184 INFO:scrape: 2023-10-15T07:44:24.185 INFO:scrape:Crash: Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' 2023-10-15T07:44:24.185 INFO:scrape:ceph version 17.2.6-1415-g9d6d32bb (9d6d32bb3452d5179bde2ee1cfa05df7a65f4586) quincy (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55b7b5512ea7] 2: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x9fb) [0x55b7b5c12adb] 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x2df) [0x55b7b5c23c7f] 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55b7b57de083] 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7c2) [0x55b7b5a34d82] 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x50d) [0x55b7b5755c9d] 7: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xd25) [0x55b7b57b2ec5] 8: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x288d) [0x55b7b57ba52d] 9: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c0) [0x55b7b5615b40] 10: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x6d) [0x55b7b591e4ad] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x112f) [0x55b7b565081f] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b7b5d94495] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b7b5d96bb4] 14: /lib64/libpthread.so.0(+0x81ca) [0x7f2a7c2e81ca] 2023-10-15T07:44:24.185 INFO:scrape:1 jobs: ['7425732'] 2023-10-15T07:44:24.186 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{centos_8}', 'fs/full/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{ignorelist_health', 'tasks/mgr-osd-full}'] 2023-10-15T07:44:24.186 INFO:scrape: 2023-10-15T07:44:24.187 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-10-15T07:44:24.187 INFO:scrape:1 jobs: ['7425825'] 2023-10-15T07:44:24.187 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-10-15T07:44:24.188 INFO:scrape: 2023-10-15T07:44:24.188 INFO:scrape:Failure: Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes) 2023-10-15T07:44:24.189 INFO:scrape:1 jobs: ['7425801'] 2023-10-15T07:44:24.189 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-10-15T07:44:24.189 INFO:scrape: 2023-10-15T07:44:24.200 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/ failed: 19 dead: 2 running: 1 waiting: 0 queued: 0 passed: 167 Fail ================================================================= [7425732] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:31:13 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425732/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425732/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5d22ebccc48477c915bef228e2eed6e Command failed (workunit test fs/full/subvolume_snapshot_rm.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/full/subvolume_snapshot_rm.sh' [7425735] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} ----------------------------------------------------------------- time: 02:00:37 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425735/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425735/ "1697307455.6124878 mon.a (mon.0) 2225 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7425746] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:05:28 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425746/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425746/ "1697308848.5385485 mon.a (mon.0) 2220 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7425747] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:14:01 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425747/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd9d44bb4a034d8498e21367c7b1037a Command failed on smithi139 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest- pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7967dbbc-6abd-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json' [7425752] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:36:24 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425752/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425752/ "1697308972.0461354 mon.a (mon.0) 2465 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log [7425770] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:58 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425770/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425770/ sentry: https://sentry.ceph.com/organizations/ceph/?query=636388f0c5ee4e4e87e9417ef2c8fdab Command failed on smithi183 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7425775] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425775/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425775/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bcd48b9f328e42279745d634679f6c9b Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 [7425778] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:23:19 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425778/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425778/ "1697309423.174042 mon.a (mon.0) 532 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7425783] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:23:05 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425783/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425783/ "1697309097.1160598 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 218088.856815 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7425784] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 00:13:33 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425784/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425784/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bafaa631f544631a13824d4e65d7550 Command failed on smithi088 with status 5: 'sudo systemctl stop ceph-17bdafae-6ac2-11ee-8db6-212e2dc638e7@mon.b' [7425801] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:23:49 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425801/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ceded98f2cc4e45b3f3d3d2f6161dd9 Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes) [7425823] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:57:56 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425823/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425823/ "1697313121.2845657 mon.a (mon.0) 2382 : cluster [WRN] Health check failed: Degraded data redundancy: 42/3232 objects degraded (1.300%), 2 pgs degraded (PG_DEGRADED)" in cluster log [7425824] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 01:48:14 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425824/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425824/ "1697312752.7010894 mon.a (mon.0) 1040 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7425825] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:35:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425825/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425825/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7c3cbd11c13c48b8a199770791504b71 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7425830] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:18:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425830/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0fc67112b1434a7fb904ec8560c0e7b8 Command failed (workunit test libcephfs/test.sh) on smithi045 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7425836] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425836/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425836/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08ad0eb74ed84914b0dccf80e5a3df9a Command failed on smithi163 with status 1: 'sudo yum install -y kernel' [7425840] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:35:18 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425840/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85db848c02b046cea3bb6b271a37233f Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays) [7425843] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:50 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425843/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6191c6fedc134055822f7afe8edf7c02 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 [7425861] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425861/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=961bf7ace7b64101b2e4d550646b3d7e Command failed on smithi098 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' Dead ================================================================= [7425834] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2d4f4eedb6b441c8ec30e077a8fea22 SSH connection to smithi163 was lost: 'sudo chmod 0666 /tmp/tmp.KWm4mPiufy' [7425835] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425835/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425835/ Error reimaging machines: Failed to power on smithi163 Running ================================================================= [7425857] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425857/ Pass ================================================================= [7425682] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:21:41 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425682/ [7425683] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:20:57 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425683/ [7425684] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:19 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425684/ [7425685] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:14 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425685/ [7425686] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:12:34 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425686/ [7425687] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:14:35 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425687/ [7425688] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:32:03 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425688/ [7425689] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:20:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425689/ [7425690] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:18:23 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425690/ [7425691] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:27:11 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425691/ [7425692] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:13:31 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425692/ [7425693] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:14:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425693/ [7425694] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:30 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425694/ [7425695] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:56:05 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425695/ [7425696] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:16:24 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425696/ [7425697] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:08 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425697/ [7425698] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:19:02 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425698/ [7425699] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:54:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425699/ [7425700] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:38:47 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425700/ [7425701] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:17:59 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425701/ [7425702] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:14:01 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425702/ [7425703] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:24 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425703/ [7425704] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:45:39 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425704/ [7425705] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:14 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425705/ [7425706] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:26:08 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425706/ [7425707] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} time: 00:40:25 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425707/ [7425708] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 00:50:33 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425708/ [7425709] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:27 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425709/ [7425710] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:30 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425710/ [7425711] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:12 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425711/ [7425712] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:30 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425712/ [7425713] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:13:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425713/ [7425714] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:28:34 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425714/ [7425715] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:20:13 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425715/ [7425716] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} time: 00:38:35 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425716/ [7425717] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:12:44 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425717/ [7425718] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:36 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425718/ [7425719] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:20 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425719/ [7425720] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:14 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425720/ [7425721] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:14:05 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425721/ [7425722] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:23:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425722/ [7425723] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:51:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425723/ [7425724] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:33:13 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425724/ [7425725] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:14:53 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425725/ [7425726] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:18:08 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425726/ [7425727] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} time: 00:27:28 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425727/ [7425728] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:20:25 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425728/ [7425729] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:07 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425729/ [7425730] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425730/ [7425731] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:45:27 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425731/ [7425733] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:28:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425733/ [7425734] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:12:17 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425734/ [7425736] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:25:12 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425736/ [7425737] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:40:52 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425737/ [7425738] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:50:04 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425738/ [7425739] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:29:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425739/ [7425740] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425740/ [7425741] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:36:09 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425741/ [7425742] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:58:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425742/ [7425743] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{rhel_8} tasks/fstop} time: 00:14:58 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425743/ [7425744] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:22:26 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425744/ [7425745] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:25:48 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425745/ [7425748] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:13:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425748/ [7425749] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:46 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425749/ [7425750] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} time: 00:31:41 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425750/ [7425751] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:13:02 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425751/ [7425753] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:33 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425753/ [7425754] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:24:33 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425754/ [7425755] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425755/ [7425756] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:24:26 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425756/ [7425757] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:02 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425757/ [7425758] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:46:26 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425758/ [7425759] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} time: 00:30:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425759/ [7425760] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:49 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425760/ [7425761] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:51:13 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425761/ [7425762] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:26:24 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425762/ [7425763] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:42 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425763/ [7425764] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425764/ [7425765] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:26 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425765/ [7425766] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:17:50 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425766/ [7425767] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:17:39 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425767/ [7425768] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:19:21 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425768/ [7425769] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:20:15 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425769/ [7425771] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:48:03 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425771/ [7425772] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:37:53 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425772/ [7425773] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 01:15:08 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425773/ [7425774] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:36:10 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425774/ [7425776] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:44:22 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425776/ [7425777] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:24:56 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425777/ [7425779] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:30:43 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425779/ [7425780] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:49:25 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425780/ [7425781] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425781/ [7425782] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:29:38 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425782/ [7425785] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:19:42 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425785/ [7425786] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:11 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425786/ [7425787] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:12:14 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425787/ [7425788] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425788/ [7425789] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:15:11 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425789/ [7425790] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:41 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425790/ [7425791] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:20 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425791/ [7425792] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:29:58 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425792/ [7425793] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:18:17 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425793/ [7425794] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:19:31 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425794/ [7425795] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:42:51 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425795/ [7425796] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:51 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425796/ [7425797] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:12 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425797/ [7425798] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:18:40 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425798/ [7425799] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:25:21 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425799/ [7425800] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:41:57 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425800/ [7425802] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:22:03 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425802/ [7425803] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:56 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425803/ [7425804] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:52:53 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425804/ [7425805] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:14:16 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425805/ [7425806] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:20:23 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425806/ [7425807] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:03 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425807/ [7425808] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:42:51 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425808/ [7425809] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:29 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425809/ [7425810] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:36:10 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425810/ [7425811] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} time: 00:58:14 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425811/ [7425812] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:13:53 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425812/ [7425813] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:31:46 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425813/ [7425814] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:52 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425814/ [7425815] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:16:37 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425815/ [7425816] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:23:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425816/ [7425817] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:28:41 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425817/ [7425818] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:20:36 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425818/ [7425819] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 00:42:24 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425819/ [7425820] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:25:40 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425820/ [7425821] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:15:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425821/ [7425822] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:21 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425822/ [7425826] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:32:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425826/ [7425827] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:52 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425827/ [7425828] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:19:26 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425828/ [7425829] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:11:46 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425829/ [7425831] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:50:15 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425831/ [7425832] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:47:49 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425832/ [7425833] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:10 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425833/ [7425837] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:19:14 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425837/ [7425838] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:24:34 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425838/ [7425839] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:13:04 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425839/ [7425841] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 01:40:36 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425841/ [7425842] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:42:03 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425842/ [7425844] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:26 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425844/ [7425845] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:20:26 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425845/ [7425846] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:28:13 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425846/ [7425847] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:45:48 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425847/ [7425848] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:18:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425848/ [7425849] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:16:15 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425849/ [7425850] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:12:03 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425850/ [7425851] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:39 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425851/ [7425852] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:21:59 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425852/ [7425853] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:18:32 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425853/ [7425854] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:36 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425854/ [7425855] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:17 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425855/ [7425856] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:24:23 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425856/ [7425858] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:16:12 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425858/ [7425859] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:18:19 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425859/ [7425860] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:14:43 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425860/ [7425862] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:24:38 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425862/ [7425863] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:32:17 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425863/ [7425864] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:16:18 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425864/ [7425865] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:30:48 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425865/ [7425866] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 00:53:45 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425866/ [7425867] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:24:02 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425867/ [7425868] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:30:18 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425868/ [7425869] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:12:43 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425869/ [7425870] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:14:45 info: http://pulpito.ceph.com/yuriw-2023-10-13_20:07:38-fs-quincy-release-distro-default-smithi/7425870/