2022-08-19T07:25:03.462 INFO:root:teuthology version: 1.1.1.dev412+gb1d387f1
2022-08-19T07:25:03.463 INFO:teuthology.results:subset: '31/32'
2022-08-19T07:25:03.464 INFO:teuthology.results:seed: '1073'
2022-08-19T10:54:22.322 INFO:root:teuthology version: 1.1.1.dev412+gb1d387f1
2022-08-19T10:54:22.352 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2022-08-19T16:38:52.852 INFO:teuthology.results:Tests finished! gathering results...
2022-08-19T16:39:12.195 INFO:scrape:Found 164 jobs
2022-08-19T16:39:38.184 INFO:scrape:Found 10 distinct failure reasons
2022-08-19T16:39:38.185 INFO:scrape:Failure: "2022-08-19T12:53:32.076754+0000 osd.0 (osd.0) 10 : cluster [ERR] 5.1es0 deep-scrub : stat mismatch, got 21/20 objects, 0/0 clones, 21/20 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/0 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
2022-08-19T16:39:38.186 INFO:scrape:3 jobs: ['6980333', '6980424', '6980440']
2022-08-19T16:39:38.186 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-08-19T16:39:38.187 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-08-19T16:39:38.187 INFO:scrape:
2022-08-19T16:39:38.188 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi141 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'
2022-08-19T16:39:38.188 INFO:scrape:1 jobs: ['6980407']
2022-08-19T16:39:38.189 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}']
2022-08-19T16:39:38.189 INFO:scrape:
2022-08-19T16:39:38.190 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds
2022-08-19T16:39:38.190 INFO:scrape:4 jobs: ['6980337', '6980467', '6980344', '6980379']
2022-08-19T16:39:38.191 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}']
2022-08-19T16:39:38.191 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2022-08-19T16:39:38.192 INFO:scrape:
2022-08-19T16:39:38.192 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2022-08-19T16:39:38.193 INFO:scrape:9 jobs: ['6980361', '6980404', '6980374', '6980321', '6980423', '6980453', '6980438', '6980341', '6980391']
2022-08-19T16:39:38.193 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{frag', 'scrub/no', 'session_timeout}', 'standby-replay', 'tasks/{0-check-counter', 'workunit/suites/pjd}}']
2022-08-19T16:39:38.194 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'scrub/no', 'session_timeout}', 'standby-replay', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-no-extra-options}', 'subvolume/{with-quota}', 'tasks/{0-check-counter', 'workunit/suites/pjd}}', 'wsync/yes}']
2022-08-19T16:39:38.194 INFO:scrape:
2022-08-19T16:39:38.195 INFO:scrape:Failure: Test failure: test_subvolume_group_ls_filter_internal_directories (tasks.cephfs.test_volumes.TestSubvolumeGroups)
2022-08-19T16:39:38.196 INFO:scrape:1 jobs: ['6980381']
2022-08-19T16:39:38.196 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}']
2022-08-19T16:39:38.197 INFO:scrape:
2022-08-19T16:39:38.197 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)
2022-08-19T16:39:38.198 INFO:scrape:1 jobs: ['6980392']
2022-08-19T16:39:38.198 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}']
2022-08-19T16:39:38.199 INFO:scrape:
2022-08-19T16:39:38.199 INFO:scrape:Failure: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)
2022-08-19T16:39:38.199 INFO:scrape:1 jobs: ['6980442']
2022-08-19T16:39:38.200 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/pool-perm}']
2022-08-19T16:39:38.200 INFO:scrape:
2022-08-19T16:39:38.201 INFO:scrape:Failure: "2022-08-19T11:03:25.555900+0000 mon.a (mon.0) 351 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log
2022-08-19T16:39:38.201 INFO:scrape:3 jobs: ['6980462', '6980319', '6980338']
2022-08-19T16:39:38.202 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health']
2022-08-19T16:39:38.202 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/pool-perm}', 'tasks/snap-schedule}']
2022-08-19T16:39:38.203 INFO:scrape:
2022-08-19T16:39:38.203 INFO:scrape:Failure: Test failure: test_table_tool (tasks.cephfs.test_journal_repair.TestJournalRepair)
2022-08-19T16:39:38.204 INFO:scrape:1 jobs: ['6980421']
2022-08-19T16:39:38.204 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/journal-repair}']
2022-08-19T16:39:38.205 INFO:scrape:
2022-08-19T16:39:38.205 INFO:scrape:Failure: "2022-08-19T11:09:09.761257+0000 mds.e (mds.1) 26 : cluster [WRN] client.4580 isn't responding to mclientcaps(revoke), ino 0x2000000116f pending pAsLsXsFscr issued pAsLsXsFscrb, sent 300.004136 seconds ago" in cluster log
2022-08-19T16:39:38.205 INFO:scrape:1 jobs: ['6980458']
2022-08-19T16:39:38.206 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health}']
2022-08-19T16:39:38.207 INFO:scrape:
2022-08-19T16:39:38.215 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/
failed:  25
dead:    0
running: 0
waiting: 0
queued:  0
passed:  139


Fail
=================================================================
[6980319]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:12:38
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980319/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980319/

    "2022-08-19T07:48:05.068837+0000 mon.a (mon.0) 224 : cluster [WRN] Health
    check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1
    pg degraded (PG_DEGRADED)" in cluster log

[6980321]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:39:53
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980321/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980321/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fbe0886f2d24407082aaae6ce2df02bd

    Command failed (workunit test suites/pjd.sh) on smithi103 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980333]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   05:36:57
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980333/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980333/

    "2022-08-19T12:53:32.076754+0000 osd.0 (osd.0) 10 : cluster [ERR] 5.1es0
    deep-scrub : stat mismatch, got 21/20 objects, 0/0 clones, 21/20 dirty, 0/0
    omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/0 bytes, 0/0
    manifest objects, 0/0 hit_set_archive bytes." in cluster log

[6980337]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:21:51
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980337/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980337/
sentry: https://sentry.ceph.com/organizations/ceph/?query=44bc9a588a5f46afb785fa010cc8aca3

    reached maximum tries (90) after waiting for 540 seconds

[6980338]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:47:18
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980338/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980338/

    "2022-08-19T08:00:44.163126+0000 mon.a (mon.0) 423 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6980341]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:26:12
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980341/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980341/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8388d5680d6b448bac42006c44278412

    Command failed (workunit test suites/pjd.sh) on smithi109 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980344]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:21:34
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980344/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980344/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a6d84b28b5874cbab1752b25a504f8d5

    reached maximum tries (90) after waiting for 540 seconds

[6980361]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:37:53
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980361/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980361/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1341a693c7744f1c9916015afd455f12

    Command failed (workunit test suites/pjd.sh) on smithi057 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980374]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:26:08
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980374/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980374/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ed46c453d154420cabb959a81092e5a6

    Command failed (workunit test suites/pjd.sh) on smithi017 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980379]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:21:20
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980379/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980379/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a31af653ac94cc68de22d518d596c32

    reached maximum tries (90) after waiting for 540 seconds

[6980381]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}
-----------------------------------------------------------------
time:   00:30:16
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980381/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980381/
sentry: https://sentry.ceph.com/organizations/ceph/?query=7bc128dbdc6048a08e6aba953905391c

    Test failure: test_subvolume_group_ls_filter_internal_directories
    (tasks.cephfs.test_volumes.TestSubvolumeGroups)

[6980391]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:35:45
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980391/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980391/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d6c7b43386d54b8c9717554b4ff975ef

    Command failed (workunit test suites/pjd.sh) on smithi101 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980392]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits}
-----------------------------------------------------------------
time:   00:13:12
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980392/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980392/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5ed6a8ac8172403291c08bf9611e020b

    Test failure: test_client_cache_size
    (tasks.cephfs.test_client_limits.TestClientLimits)

[6980404]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:26:14
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980404/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980404/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2df082f9aa894210a6d259d0df884af4

    Command failed (workunit test suites/pjd.sh) on smithi138 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980407]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}}
-----------------------------------------------------------------
time:   00:14:39
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980407/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980407/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8a598613e88944ffaedde0ffe9a7cff6

    Command failed (workunit test libcephfs/test.sh) on smithi141 with status
    139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'

[6980421]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair}
-----------------------------------------------------------------
time:   00:17:26
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980421/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980421/
sentry: https://sentry.ceph.com/organizations/ceph/?query=09fe2c2b6e0542cf85c65eb8698cfa60

    Test failure: test_table_tool
    (tasks.cephfs.test_journal_repair.TestJournalRepair)

[6980423]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:36:42
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980423/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980423/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cbfab82cf0aa4d49986c4e015e75b9c0

    Command failed (workunit test suites/pjd.sh) on smithi114 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980424]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:21:31
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980424/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980424/

    "2022-08-19T12:54:05.600368+0000 osd.7 (osd.7) 14 : cluster [ERR] 5.1as0
    deep-scrub : stat mismatch, got 0/4 objects, 0/0 clones, 0/4 dirty, 0/0
    omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/41591 bytes, 0/0
    manifest objects, 0/0 hit_set_archive bytes." in cluster log

[6980438]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:25:43
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980438/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980438/
sentry: https://sentry.ceph.com/organizations/ceph/?query=1722dfdad03449b09fc600583cbebee1

    Command failed (workunit test suites/pjd.sh) on smithi138 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980440]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   03:32:17
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980440/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980440/

    "2022-08-19T13:16:12.703607+0000 osd.4 (osd.4) 52 : cluster [ERR] 5.14s0
    deep-scrub : stat mismatch, got 0/1 objects, 0/1 clones, 0/1 dirty, 0/0
    omap, 0/0 pinned, 0/0 hit_set_archive, 0/1 whiteouts, 0/0 bytes, 0/0
    manifest objects, 0/0 hit_set_archive bytes." in cluster log

[6980442]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}
-----------------------------------------------------------------
time:   00:19:34
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980442/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980442/
sentry: https://sentry.ceph.com/organizations/ceph/?query=db1f66717201439c878597f5c62cc860

    Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

[6980453]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:44:34
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980453/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980453/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9ec97ea78289468fa155ace26584d95d

    Command failed (workunit test suites/pjd.sh) on smithi141 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c7606b2c8bdef281f7eee6b171bb9adc82612b43
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[6980458]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   06:01:02
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980458/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980458/

    "2022-08-19T11:09:09.761257+0000 mds.e (mds.1) 26 : cluster [WRN]
    client.4580 isn't responding to mclientcaps(revoke), ino 0x2000000116f
    pending pAsLsXsFscr issued pAsLsXsFscrb, sent 300.004136 seconds ago" in
    cluster log

[6980462]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}
-----------------------------------------------------------------
time:   00:55:02
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980462/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980462/

    "2022-08-19T11:03:25.555900+0000 mon.a (mon.0) 351 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in
    cluster log

[6980467]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:19:33
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980467/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980467/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e6e642bcfa5346cc8d0cc3f8b91dd7a1

    reached maximum tries (90) after waiting for 540 seconds



Pass
=================================================================
[6980316] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:28:54
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980316/

[6980317] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:22:15
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980317/

[6980318] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   02:02:54
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980318/

[6980320] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:27:19
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980320/

[6980322] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:17:18
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980322/

[6980323] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:15:56
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980323/

[6980324] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit}
time:   00:37:25
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980324/

[6980325] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:21:21
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980325/

[6980326] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:29:17
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980326/

[6980327] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
time:   02:20:45
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980327/

[6980328] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind}
time:   01:37:27
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980328/

[6980329] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:14:15
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980329/

[6980330] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:59:54
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980330/

[6980331] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:46:13
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980331/

[6980332] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:15
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980332/

[6980334] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:38:28
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980334/

[6980335] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:28:47
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980335/

[6980336] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:21:43
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980336/

[6980339] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:26:06
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980339/

[6980340] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:19:11
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980340/

[6980342] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:24:39
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980342/

[6980343] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:16
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980343/

[6980345] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:56
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980345/

[6980346] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:54
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980346/

[6980347] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays}
time:   00:32:30
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980347/

[6980348] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/dbench}}
time:   01:51:02
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980348/

[6980349] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:23:59
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980349/

[6980350] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:21:48
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980350/

[6980351] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:43
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980351/

[6980352] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:17:33
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980352/

[6980353] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:27:22
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980353/

[6980354] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:34:56
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980354/

[6980355] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:27:10
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980355/

[6980356] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:03:40
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980356/

[6980357] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:31:25
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980357/

[6980358] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:36:30
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980358/

[6980359] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:16:42
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980359/

[6980360] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:33:53
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980360/

[6980362] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries}
time:   00:21:18
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980362/

[6980363] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:30:17
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980363/

[6980364] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:01
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980364/

[6980365] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:16:18
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980365/

[6980366] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:23:21
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980366/

[6980367] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:09
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980367/

[6980368] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:37:18
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980368/

[6980369] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:23:06
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980369/

[6980370] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   03:08:35
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980370/

[6980371] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:25:57
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980371/

[6980372] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:30:52
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980372/

[6980373] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:17:14
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980373/

[6980375] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:38:38
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980375/

[6980376] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:33:19
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980376/

[6980377] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:15:48
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980377/

[6980378] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:45:14
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980378/

[6980380] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
time:   01:26:41
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980380/

[6980382] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:04:56
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980382/

[6980383] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:26:44
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980383/

[6980384] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:28:39
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980384/

[6980385] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:22:33
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980385/

[6980386] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:21:46
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980386/

[6980387] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:57:56
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980387/

[6980388] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:23
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980388/

[6980389] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:15:21
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980389/

[6980390] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:20
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980390/

[6980393] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/direct_io}}
time:   00:33:08
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980393/

[6980394] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:23:26
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980394/

[6980395] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:34:44
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980395/

[6980396] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:28:32
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980396/

[6980397] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:22:32
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980397/

[6980398] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:18:52
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980398/

[6980399] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:39:55
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980399/

[6980400] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:59:22
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980400/

[6980401] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:28:22
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980401/

[6980402] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:22:48
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980402/

[6980403] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:10:16
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980403/

[6980405] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:20:11
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980405/

[6980406] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:39
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980406/

[6980408] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   00:50:24
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980408/

[6980409] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:42
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980409/

[6980410] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:27:40
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980410/

[6980411] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:30:11
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980411/

[6980412] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:48
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980412/

[6980413] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:34:21
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980413/

[6980414] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:26:09
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980414/

[6980415] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:48
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980415/

[6980416] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:27:37
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980416/

[6980417] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:24:05
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980417/

[6980418] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:21:27
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980418/

[6980419] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:41:07
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980419/

[6980420] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:32:40
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980420/

[6980422] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:05
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980422/

[6980425] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/misc}}
time:   00:43:43
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980425/

[6980426] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:22:32
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980426/

[6980427] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:18:44
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980427/

[6980428] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:22:49
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980428/

[6980429] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:42:12
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980429/

[6980430] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:28:54
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980430/

[6980431] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:39:17
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980431/

[6980432] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:21:51
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980432/

[6980433] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:15:04
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980433/

[6980434] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:41:05
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980434/

[6980435] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
time:   01:30:50
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980435/

[6980436] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:25:57
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980436/

[6980437] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:29:55
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980437/

[6980439] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:22:01
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980439/

[6980441] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:22:41
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980441/

[6980443] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:22:52
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980443/

[6980444] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:58:09
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980444/

[6980445] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:41
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980445/

[6980446] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
time:   00:11:54
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980446/

[6980447] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:20:45
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980447/

[6980448] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:26:34
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980448/

[6980449] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:26:16
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980449/

[6980450] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsx}}
time:   00:45:48
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980450/

[6980451] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:14:33
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980451/

[6980452] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:23:05
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980452/

[6980454] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:33:03
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980454/

[6980455] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:29:13
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980455/

[6980456] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:08
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980456/

[6980457] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:16:24
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980457/

[6980459] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:22:14
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980459/

[6980460] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:11:43
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980460/

[6980461] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:33:46
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980461/

[6980463] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:15:40
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980463/

[6980464] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:38:50
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980464/

[6980465] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:50:57
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980465/

[6980466] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}}
time:   00:35:20
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980466/

[6980468] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:30:12
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980468/

[6980469] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:42:11
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980469/

[6980470] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:18:37
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980470/

[6980471] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:16:44
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980471/

[6980472] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}}
time:   00:21:21
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980472/

[6980473] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:24:49
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980473/

[6980474] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:09:08
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980474/

[6980475] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:25:08
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980475/

[6980476] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration}
time:   00:18:13
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980476/

[6980477] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:15:35
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980477/

[6980478] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   04:25:55
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980478/

[6980479] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:21:43
info:   http://pulpito.ceph.com/vshankar-2022-08-19_06:21:01-fs-wip-vshankar-testing1-20220818-082047-testing-default-smithi/6980479/