2021-01-21T12:57:25.814 INFO:root:teuthology version: 1.0.0-0746baa9
2021-01-21T12:57:25.815 INFO:teuthology.results:seed: '6481'
2021-01-21T14:11:49.627 INFO:root:teuthology version: 1.0.0-0746baa9
2021-01-21T14:11:49.677 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-01-22T01:50:11.541 INFO:teuthology.results:Tests finished! gathering results...
2021-01-22T01:50:28.409 INFO:scrape:Found 100 jobs
2021-01-22T01:50:36.621 INFO:scrape:Found 7 distinct failure reasons
2021-01-22T01:50:36.623 INFO:scrape:Failure: "2021-01-21T14:20:58.797580+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000030a8 (/client.0/tmp/tmp.08kK5671Tc/p6) see mds.f log and `damage ls` output for details" in cluster log
2021-01-22T01:50:36.624 INFO:scrape:7 jobs: ['5807306', '5807276', '5807295', '5807321', '5807273', '5807299', '5807316']
2021-01-22T01:50:36.625 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:50:36.626 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsstress}}', 'workunit/suites/pjd}}']
2021-01-22T01:50:36.627 INFO:scrape:
2021-01-22T01:50:36.628 INFO:scrape:Crash: Found coredumps on ubuntu@smithi050.front.sepia.ceph.com
2021-01-22T01:50:36.629 INFO:scrape:ceph version 16.0.0-9201-gc0e5d660 (c0e5d66088f5063a44e1956812448b957f477085) pacific (dev)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7fd28a7bdb20]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fd28ba1173b]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x276904) [0x7fd28ba11904]
 6: (ProtocolV1::handle_message_footer(char*, int)+0xebe) [0x7fd28bd03e7e]
 7: /usr/lib64/ceph/libceph-common.so.2(+0x56085d) [0x7fd28bcfb85d]
 8: (AsyncConnection::process()+0x789) [0x7fd28bce9df9]
 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xcb7) [0x7fd28bd44697]
 10: /usr/lib64/ceph/libceph-common.so.2(+0x5afbac) [0x7fd28bd4abac]
 11: /lib64/libstdc++.so.6(+0xc2ba3) [0x7fd289bf6ba3]
 12: /lib64/libpthread.so.0(+0x814a) [0x7fd28a7b314a]
 13: clone()
2021-01-22T01:50:36.629 INFO:scrape:1 jobs: ['5807314']
2021-01-22T01:50:36.631 INFO:scrape:suites: ['2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/3', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:50:36.632 INFO:scrape:
2021-01-22T01:50:36.633 INFO:scrape:Dead: b'2021-01-22T01:47:39.944 DEBUG:teuthology.exit:Finished running handlers'
2021-01-22T01:50:36.634 INFO:scrape:1 jobs: ['5807292']
2021-01-22T01:50:36.635 INFO:scrape:suites: ['2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:50:36.636 INFO:scrape:
2021-01-22T01:50:36.637 INFO:scrape:Crash: Command failed (workunit test suites/fsstress.sh) on smithi025 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c0e5d66088f5063a44e1956812448b957f477085 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2021-01-22T01:50:36.638 INFO:scrape:ceph version 16.0.0-9201-gc0e5d660 (c0e5d66088f5063a44e1956812448b957f477085) pacific (dev)
 1: /lib64/libpthread.so.0(+0x12dc0) [0xf9bddc0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x50b9a23]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x27bbec) [0x50b9bec]
 6: (Monitor::~Monitor()+0xc2d) [0x3e879d]
 7: (Monitor::~Monitor()+0xd) [0x3e87fd]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-01-22T01:50:36.639 INFO:scrape:1 jobs: ['5807293']
2021-01-22T01:50:36.639 INFO:scrape:suites: ['centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:50:36.640 INFO:scrape:
2021-01-22T01:50:36.642 INFO:scrape:Failure: "2021-01-21T13:33:21.110197+0000 mds.e (mds.0) 14 : cluster [WRN] Scrub error on inode 0x100000001fb (/client.0/tmp/fsstress/ltp-full-20091231/testcases/ballista) see mds.e log and `damage ls` output for details" in cluster log
2021-01-22T01:50:36.643 INFO:scrape:1 jobs: ['5807255']
2021-01-22T01:50:36.644 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/fsstress}}']
2021-01-22T01:50:36.645 INFO:scrape:
2021-01-22T01:50:36.646 INFO:scrape:Failure: "2021-01-21T14:16:56.198961+0000 mon.a (mon.0) 354 : cluster [WRN] Health check failed: Degraded data redundancy: 445/13424 objects degraded (3.315%), 5 pgs degraded (PG_DEGRADED)" in cluster log
2021-01-22T01:50:36.647 INFO:scrape:1 jobs: ['5807252']
2021-01-22T01:50:36.648 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}}']
2021-01-22T01:50:36.649 INFO:scrape:
2021-01-22T01:50:36.650 INFO:scrape:Failure: "2021-01-21T13:36:52.992059+0000 mds.a (mds.2) 4 : cluster [WRN] Scrub error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see mds.a log and `damage ls` output for details" in cluster log
2021-01-22T01:50:36.651 INFO:scrape:1 jobs: ['5807245']
2021-01-22T01:50:36.651 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/3', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}']
2021-01-22T01:50:36.653 INFO:scrape:
2021-01-22T01:50:36.661 INFO:teuthology.results:Sending results to jcollin@redhat.com: Test Run: jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/
failed:  12
dead:    1
running: 0
waiting: 0
queued:  0
passed:  87


Fail
=================================================================
[5807245]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:39:44
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807245/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807245/

    "2021-01-21T13:36:52.992059+0000 mds.a (mds.2) 4 : cluster [WRN] Scrub
    error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see
    mds.a log and `damage ls` output for details" in cluster log

[5807252]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:44:18
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807252/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807252/

    "2021-01-21T14:16:56.198961+0000 mon.a (mon.0) 354 : cluster [WRN] Health
    check failed: Degraded data redundancy: 445/13424 objects degraded
    (3.315%), 5 pgs degraded (PG_DEGRADED)" in cluster log

[5807255]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:34
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807255/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807255/

    "2021-01-21T13:33:21.110197+0000 mds.e (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fb (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/ballista) see mds.e log and `damage ls` output for
    details" in cluster log

[5807273]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:36:46
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807273/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807273/

    "2021-01-21T13:53:00.087669+0000 mds.c (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.c
    log and `damage ls` output for details" in cluster log

[5807276]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:51:38
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807276/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807276/

    "2021-01-21T14:09:38.683893+0000 mds.a (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.a log and `damage
    ls` output for details" in cluster log

[5807293]  fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:12:45
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807293/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807293/
sentry: https://sentry.ceph.com/organizations/ceph/?query=acdea208d7504bf9bf29a3f0569dea5e

    Command failed (workunit test suites/fsstress.sh) on smithi025 with status
    135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=c0e5d66088f5063a44e1956812448b957f477085
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[5807295]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:29:11
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807295/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807295/

    "2021-01-21T14:12:34.739277+0000 mds.a (mds.4) 10 : cluster [WRN] Scrub
    error on inode 0x10000000264 (/client.0/tmp/testdir/dir1/dir2) see mds.a
    log and `damage ls` output for details" in cluster log

[5807299]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:20:55
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807299/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807299/

    "2021-01-21T14:18:19.350365+0000 mds.a (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x10000001aee
    (/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-48) see mds.a log and
    `damage ls` output for details" in cluster log

[5807306]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:35
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807306/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807306/

    "2021-01-21T14:20:58.797580+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x100000030a8 (/client.0/tmp/tmp.08kK5671Tc/p6) see mds.f
    log and `damage ls` output for details" in cluster log

[5807314]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:38:40
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807314/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807314/

    Found coredumps on ubuntu@smithi050.front.sepia.ceph.com

[5807316]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:11:41
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807316/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807316/

    "2021-01-21T14:35:07.438749+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.f log and `damage
    ls` output for details" in cluster log

[5807321]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:30:29
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807321/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807321/

    "2021-01-21T14:35:11.369793+0000 mds.c (mds.1) 42 : cluster [WRN] Scrub
    error on inode 0x10000000264 (/client.0/tmp/testdir/dir1/dir2) see mds.c
    log and `damage ls` output for details" in cluster log



Dead
=================================================================
[5807292]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807292/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807292/



Pass
=================================================================
[5807227] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:13:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807227/

[5807228] fs/bugs/client_trim_caps/{begin clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i22073}
time:   00:07:18
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807228/

[5807229] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:09:13
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807229/

[5807230] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:07:01
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807230/

[5807231] fs/mirror/{begin cephfs-mirror/{one-per-cluster} clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror}
time:   00:47:24
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807231/

[5807232] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:29:11
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807232/

[5807233] fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   00:20:28
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807233/

[5807234] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:38:47
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807234/

[5807235] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:08:00
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807235/

[5807236] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:32:25
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807236/

[5807237] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:03:13
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807237/

[5807238] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   03:33:38
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807238/

[5807239] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{rhel_8} tasks/fstop}
time:   00:08:51
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807239/

[5807240] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:19:55
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807240/

[5807241] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   00:30:54
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807241/

[5807242] fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   00:55:10
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807242/

[5807243] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:45:45
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807243/

[5807244] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:12:32
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807244/

[5807246] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:15:20
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807246/

[5807247] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:38:27
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807247/

[5807248] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:37:15
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807248/

[5807249] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:19:39
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807249/

[5807250] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:00:19
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807250/

[5807251] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:03
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807251/

[5807253] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:34
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807253/

[5807254] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:15:35
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807254/

[5807256] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:09:49
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807256/

[5807257] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:36:49
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807257/

[5807258] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:00
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807258/

[5807259] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:59
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807259/

[5807260] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:25
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807260/

[5807261] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:11:33
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807261/

[5807262] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:24:08
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807262/

[5807263] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:55:31
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807263/

[5807264] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:37:01
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807264/

[5807265] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:10:28
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807265/

[5807266] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:11:54
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807266/

[5807267] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:13:29
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807267/

[5807268] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:44
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807268/

[5807269] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:09:54
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807269/

[5807270] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
time:   00:43:24
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807270/

[5807271] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   01:28:41
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807271/

[5807272] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:18:56
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807272/

[5807274] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:14:30
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807274/

[5807275] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:51:00
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807275/

[5807277] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:36:54
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807277/

[5807278] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:15:33
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807278/

[5807279] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:42:59
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807279/

[5807280] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:14:33
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807280/

[5807281] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:15:34
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807281/

[5807282] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:26:01
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807282/

[5807283] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:18:23
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807283/

[5807284] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:40
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807284/

[5807285] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:09:52
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807285/

[5807286] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:20:01
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807286/

[5807287] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:56:32
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807287/

[5807288] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:12:26
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807288/

[5807289] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:17:49
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807289/

[5807290] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:11:32
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807290/

[5807291] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:12:17
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807291/

[5807294] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:02
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807294/

[5807296] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:05:45
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807296/

[5807297] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:59:19
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807297/

[5807298] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:37:22
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807298/

[5807300] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
time:   00:21:23
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807300/

[5807301] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:02:51
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807301/

[5807302] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:15:19
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807302/

[5807303] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:28:46
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807303/

[5807304] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807304/

[5807305] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:16:39
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807305/

[5807307] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807307/

[5807308] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:27:02
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807308/

[5807309] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:13:26
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807309/

[5807310] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:10:58
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807310/

[5807311] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:15
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807311/

[5807312] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:01:32
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807312/

[5807313] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:34:58
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807313/

[5807315] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:11:04
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807315/

[5807317] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:18:56
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807317/

[5807318] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:10:06
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807318/

[5807319] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:14:48
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807319/

[5807320] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:09:43
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807320/

[5807322] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   02:29:23
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807322/

[5807323] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:11:40
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807323/

[5807324] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:22:03
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807324/

[5807325] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:12:19
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807325/

[5807326] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:09:17
info:   http://pulpito.ceph.com/jcollin-2021-01-21_07:46:03-fs-wip-jcollin-dispatch-throttle-test-config-distro-basic-smithi/5807326/