2023-09-27T01:08:37.636 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-27T01:08:37.637 INFO:teuthology.results:subset: '100/1024' 2023-09-27T01:08:37.637 INFO:teuthology.results:seed: '526' 2023-09-27T09:36:35.876 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-27T09:36:35.920 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-27T12:06:48.008 INFO:teuthology.results:Tests finished! gathering results... 2023-09-27T12:07:04.764 INFO:scrape:Found 204 jobs 2023-09-27T12:07:05.711 WARNING:scrape:Misunderstood line: 2023-09-27T12:07:08.892 WARNING:scrape:Misunderstood line: 2023-09-27T12:07:08.951 WARNING:scrape:Misunderstood line: 2023-09-27T12:07:14.224 INFO:scrape:Found 8 distinct failure reasons 2023-09-27T12:07:14.225 INFO:scrape:Valgrind: cephfs-mirror-client (Leak_StillReachable) 2023-09-27T12:07:14.225 INFO:scrape:1 jobs: ['7402640'] 2023-09-27T12:07:14.225 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-09-27T12:07:14.225 INFO:scrape: 2023-09-27T12:07:14.225 INFO:scrape:Crash: [Errno 104] Connection reset by peer 2023-09-27T12:07:14.225 INFO:scrape:ceph version 18.2.0-485-g317bf1d3 (317bf1d3f95ea187402bf91781395c9ffd356fa2) reef (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x7fdd5bee741b] 2: (CDentry::check_corruption(bool)+0x877) [0x5575ecbf8967] 3: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x47) [0x5575eca51ab7] 4: (EMetaBlob::add_dir_context(CDir*, int)+0x17b) [0x5575ecd5a8eb] 5: (MDCache::create_subtree_map()+0xc41) [0x5575ecac5031] 6: (MDLog::_journal_segment_subtree_map(MDSContext*)+0x4b) [0x5575eccead5b] 7: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x4e0) [0x5575ecc86540] 8: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xd75) [0x5575ec985625] 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x5575ec987a33] 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5575ec98804c] 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5575ec9718cf] 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdd5c160208] 13: (DispatchQueue::entry()+0x50f) [0x7fdd5c15d3af] 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdd5c2238f1] 15: /lib64/libpthread.so.0(+0x81cf) [0x7fdd5ac901cf] 2023-09-27T12:07:14.225 INFO:scrape:1 jobs: ['7402725'] 2023-09-27T12:07:14.226 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-09-27T12:07:14.226 INFO:scrape: 2023-09-27T12:07:14.226 INFO:scrape:Failure: Command failed on smithi125 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-09-27T12:07:14.226 INFO:scrape:4 jobs: ['7402801', '7402702', '7402671', '7402768'] 2023-09-27T12:07:14.226 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-09-27T12:07:14.226 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-09-27T12:07:14.227 INFO:scrape: 2023-09-27T12:07:14.227 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=317bf1d3f95ea187402bf91781395c9ffd356fa2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-09-27T12:07:14.227 INFO:scrape:1 jobs: ['7402699'] 2023-09-27T12:07:14.227 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}'] 2023-09-27T12:07:14.227 INFO:scrape: 2023-09-27T12:07:14.227 INFO:scrape:Failure: "2023-09-27T04:54:44.283390+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 139460.515590 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-09-27T12:07:14.228 INFO:scrape:1 jobs: ['7402697'] 2023-09-27T12:07:14.228 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-09-27T12:07:14.228 INFO:scrape: 2023-09-27T12:07:14.228 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-09-27T12:07:14.228 INFO:scrape:1 jobs: ['7402821'] 2023-09-27T12:07:14.228 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/3', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-09-27T12:07:14.228 INFO:scrape: 2023-09-27T12:07:14.228 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds 2023-09-27T12:07:14.229 INFO:scrape:3 jobs: ['7402668', '7402650', '7402817'] 2023-09-27T12:07:14.229 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-09-27T12:07:14.229 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-09-27T12:07:14.229 INFO:scrape: 2023-09-27T12:07:14.229 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2023-09-27T12:07:14.229 INFO:scrape:1 jobs: ['7402629'] 2023-09-27T12:07:14.229 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2023-09-27T12:07:14.229 INFO:scrape: 2023-09-27T12:07:14.238 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/ failed: 13 dead: 0 running: 0 waiting: 0 queued: 0 passed: 191 Fail ================================================================= [7402629] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:17:16 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402629/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfd5a5e683fb4f88a53f0fe1528154f9 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7402640] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:24:44 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402640/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402640/ saw valgrind issues [7402650] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 02:01:33 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402650/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402650/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7402668] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:02:25 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402668/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402668/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7402671] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:02 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402671/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402671/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4009fcc8fb5a46e39dd98a05b4b34ce5 Command failed on smithi125 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7402697] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:25:25 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402697/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402697/ "2023-09-27T04:54:44.283390+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 139460.515590 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7402699] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:59:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402699/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402699/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8fb102655363485f9f835c6aaa6da678 Command failed (workunit test suites/ffsb.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=317bf1d3f95ea187402bf91781395c9ffd356fa2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7402702] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402702/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402702/ sentry: https://sentry.ceph.com/organizations/ceph/?query=526c7245f7924104b90f9a2aa1af5bd2 Command failed on smithi158 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7402725] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:35:20 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402725/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402725/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e6781f614a04a3c981b487a112acb4d [Errno 104] Connection reset by peer [7402768] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:26 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402768/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402768/ sentry: https://sentry.ceph.com/organizations/ceph/?query=396dc7c469454bf581ee5a38e31454ba Command failed on smithi064 with status 8: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7402801] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:24 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402801/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad3a0b716e8a437394c08e656ae98cd8 Command failed on smithi125 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7402817] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:39:58 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402817/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402817/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7402821] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 02:00:43 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402821/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402821/ sentry: https://sentry.ceph.com/organizations/ceph/?query=493df60a14184176b06129700f806762 saw valgrind issues Pass ================================================================= [7402622] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:23:54 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402622/ [7402623] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:29:46 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402623/ [7402624] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:29:42 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402624/ [7402625] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:14:54 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402625/ [7402626] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:17 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402626/ [7402627] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:15:55 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402627/ [7402628] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:51:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402628/ [7402630] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:25:44 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402630/ [7402631] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:15:51 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402631/ [7402632] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_latest} tasks/mirror} time: 02:19:37 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402632/ [7402633] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} time: 00:30:47 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402633/ [7402634] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402634/ [7402635] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:56:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402635/ [7402636] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:22 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402636/ [7402637] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:49:50 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402637/ [7402638] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:07:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402638/ [7402639] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 00:18:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402639/ [7402641] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:55:05 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402641/ [7402642] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:38:26 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402642/ [7402643] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:13 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402643/ [7402644] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:15:56 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402644/ [7402645] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:16:00 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402645/ [7402646] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:16:27 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402646/ [7402647] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:18:09 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402647/ [7402648] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:16:44 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402648/ [7402649] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:20:08 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402649/ [7402651] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:21:41 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402651/ [7402652] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:24 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402652/ [7402653] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402653/ [7402654] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:15:59 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402654/ [7402655] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 01:03:37 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402655/ [7402656] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:22:41 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402656/ [7402657] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:23 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402657/ [7402658] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:47:20 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402658/ [7402659] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:53:06 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402659/ [7402660] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:21:00 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402660/ [7402661] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:44 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402661/ [7402662] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:49 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402662/ [7402663] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:59:10 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402663/ [7402664] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:19:58 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402664/ [7402665] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:25:06 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402665/ [7402666] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:56:33 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402666/ [7402667] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:19 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402667/ [7402669] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:41:31 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402669/ [7402670] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:44:25 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402670/ [7402672] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:28:08 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402672/ [7402673] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:52:14 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402673/ [7402674] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:22:09 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402674/ [7402675] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:59:02 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402675/ [7402676] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:42:48 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402676/ [7402677] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} time: 01:10:39 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402677/ [7402678] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:23 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402678/ [7402679] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:18:16 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402679/ [7402680] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} time: 00:44:05 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402680/ [7402681] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:37:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402681/ [7402682] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:22:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402682/ [7402683] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:19:13 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402683/ [7402684] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:27 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402684/ [7402685] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:04 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402685/ [7402686] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:14:58 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402686/ [7402687] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:44 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402687/ [7402688] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:24:45 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402688/ [7402689] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:30 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402689/ [7402690] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} time: 01:10:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402690/ [7402691] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:20:41 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402691/ [7402692] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:14:13 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402692/ [7402693] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:39:30 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402693/ [7402694] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:57 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402694/ [7402695] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:45:35 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402695/ [7402696] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:25:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402696/ [7402698] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:38:34 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402698/ [7402700] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:52:05 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402700/ [7402701] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:17:59 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402701/ [7402703] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:49:45 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402703/ [7402704] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:46:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402704/ [7402705] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:41 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402705/ [7402706] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:21:37 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402706/ [7402707] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:33:37 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402707/ [7402708] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:20:58 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402708/ [7402709] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402709/ [7402710] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:27:59 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402710/ [7402711] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:34 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402711/ [7402712] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:47 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402712/ [7402713] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:25:03 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402713/ [7402714] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:28:16 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402714/ [7402715] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 01:54:18 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402715/ [7402716] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:27:31 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402716/ [7402717] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:16:19 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402717/ [7402718] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:25:49 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402718/ [7402719] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:13 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402719/ [7402720] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:14:42 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402720/ [7402721] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} time: 00:40:01 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402721/ [7402722] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:22:40 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402722/ [7402723] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:40 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402723/ [7402724] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} time: 00:18:16 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402724/ [7402726] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:55:47 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402726/ [7402727] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:15:59 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402727/ [7402728] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} time: 00:45:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402728/ [7402729] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:39:08 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402729/ [7402730] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:55:46 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402730/ [7402731] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:43 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402731/ [7402732] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:29:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402732/ [7402733] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:35 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402733/ [7402734] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:23:55 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402734/ [7402735] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:19:18 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402735/ [7402736] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:02:59 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402736/ [7402737] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:46:08 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402737/ [7402738] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:24 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402738/ [7402739] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:43:31 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402739/ [7402740] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:15 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402740/ [7402741] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:42:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402741/ [7402742] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:20:19 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402742/ [7402743] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:45 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402743/ [7402744] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:35:06 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402744/ [7402745] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:39:29 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402745/ [7402746] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:55:31 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402746/ [7402747] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:53:42 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402747/ [7402748] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:24:36 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402748/ [7402749] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:17:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402749/ [7402750] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:14:36 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402750/ [7402751] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-workload/new_ops/{0-clients/kclient 1-client-sanity}}} time: 00:14:29 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402751/ [7402752] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:31:34 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402752/ [7402753] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:35:59 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402753/ [7402754] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:28:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402754/ [7402755] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:21:20 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402755/ [7402756] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:32:09 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402756/ [7402757] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:34 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402757/ [7402758] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:35:28 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402758/ [7402759] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:08 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402759/ [7402760] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:22 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402760/ [7402761] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:53 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402761/ [7402762] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:21:56 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402762/ [7402763] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:20:49 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402763/ [7402764] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:41:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402764/ [7402765] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:52 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402765/ [7402766] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:19:33 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402766/ [7402767] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:18:44 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402767/ [7402769] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:49:35 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402769/ [7402770] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:47 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402770/ [7402771] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:46:39 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402771/ [7402772] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:46:43 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402772/ [7402773] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:43:10 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402773/ [7402774] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:42 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402774/ [7402775] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:50:00 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402775/ [7402776] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:17:03 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402776/ [7402777] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:35:43 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402777/ [7402778] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} time: 00:15:32 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402778/ [7402779] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:26:52 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402779/ [7402780] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:24 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402780/ [7402781] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:18:40 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402781/ [7402782] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:30:34 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402782/ [7402783] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} time: 00:22:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402783/ [7402784] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} time: 00:38:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402784/ [7402785] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:27:24 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402785/ [7402786] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:19 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402786/ [7402787] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:19:15 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402787/ [7402788] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:53 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402788/ [7402789] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:17:03 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402789/ [7402790] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:01:17 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402790/ [7402791] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:34:06 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402791/ [7402792] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:14:35 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402792/ [7402793] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:44:37 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402793/ [7402794] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:22:23 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402794/ [7402795] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 00:59:32 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402795/ [7402796] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} time: 00:34:45 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402796/ [7402797] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:28:19 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402797/ [7402798] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:12 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402798/ [7402799] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:22:45 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402799/ [7402800] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:16:07 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402800/ [7402802] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:34:36 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402802/ [7402803] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:57:37 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402803/ [7402804] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:22:14 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402804/ [7402805] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} time: 00:48:58 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402805/ [7402806] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:13:47 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402806/ [7402807] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:02 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402807/ [7402808] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:37:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402808/ [7402809] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:21:39 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402809/ [7402810] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:23:46 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402810/ [7402811] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:23 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402811/ [7402812] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:19:26 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402812/ [7402813] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:49 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402813/ [7402814] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:24:29 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402814/ [7402815] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:44 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402815/ [7402816] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:16:05 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402816/ [7402818] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:14:09 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402818/ [7402819] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:13:45 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402819/ [7402820] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:32:21 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402820/ [7402822] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:50 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402822/ [7402823] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:36:42 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402823/ [7402824] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:56:02 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402824/ [7402825] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:19:48 info: http://pulpito.ceph.com/vshankar-2023-09-27_01:02:08-fs-wip-vshankar-testing-reef-20230926.161455-testing-default-smithi/7402825/