2021-01-20T15:51:42.864 INFO:root:teuthology version: 1.0.0-0746baa9
2021-01-20T15:51:42.866 INFO:teuthology.results:seed: '3545'
2021-01-20T18:42:35.693 INFO:root:teuthology version: 1.0.0-0746baa9
2021-01-20T18:42:35.755 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-01-21T06:08:50.904 INFO:teuthology.results:Tests finished! gathering results...
2021-01-21T06:09:05.964 INFO:scrape:Found 100 jobs
2021-01-21T06:09:17.081 INFO:scrape:Found 9 distinct failure reasons
2021-01-21T06:09:17.082 INFO:scrape:Failure: "2021-01-20T18:58:19.123081+0000 mds.c (mds.0) 14 : cluster [WRN] Scrub error on inode 0x100000012df (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_posix_testsuite) see mds.c log and `damage ls` output for details" in cluster log
2021-01-21T06:09:17.083 INFO:scrape:7 jobs: ['5805109', '5805069', '5805079', '5805048', '5805102', '5805058', '5805119']
2021-01-21T06:09:17.083 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.084 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsstress}}', 'workunit/suites/pjd}}']
2021-01-21T06:09:17.084 INFO:scrape:
2021-01-21T06:09:17.085 INFO:scrape:Failure: "2021-01-20T18:32:45.841472+0000 osd.0 (osd.0) 3 : cluster [WRN] Throttler Limit has been hit. Some message processing may be significantly delayed." in cluster log
2021-01-21T06:09:17.085 INFO:scrape:6 jobs: ['5805101', '5805051', '5805117', '5805082', '5805055', '5805067']
2021-01-21T06:09:17.086 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{frag_enable', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.086 INFO:scrape:suites union: ['2-workunit/suites/ffsb}}', 'clusters/1a5s-mds-1c-client', 'clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}}']
2021-01-21T06:09:17.087 INFO:scrape:
2021-01-21T06:09:17.087 INFO:scrape:Failure: "2021-01-20T19:19:55.533058+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000002aa2 (/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-59) see mds.f log and `damage ls` output for details" in cluster log
2021-01-21T06:09:17.088 INFO:scrape:3 jobs: ['5805076', '5805073', '5805098']
2021-01-21T06:09:17.088 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.089 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}']
2021-01-21T06:09:17.090 INFO:scrape:
2021-01-21T06:09:17.091 INFO:scrape:Crash: Command failed (workunit test suites/fsstress.sh) on smithi089 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d8fc237ed7d71190da53e81746dff3e13ae8eb56 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2021-01-21T06:09:17.091 INFO:scrape:ceph version 16.0.0-9177-gd8fc237e (d8fc237ed7d71190da53e81746dff3e13ae8eb56) pacific (dev)
 1: /lib64/libpthread.so.0(+0x12dc0) [0xf9bddc0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x50b9a23]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x27bbec) [0x50b9bec]
 6: (Monitor::~Monitor()+0xc2d) [0x3e879d]
 7: (Monitor::~Monitor()+0xd) [0x3e87fd]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-01-21T06:09:17.092 INFO:scrape:1 jobs: ['5805096']
2021-01-21T06:09:17.092 INFO:scrape:suites: ['centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.093 INFO:scrape:
2021-01-21T06:09:17.094 INFO:scrape:Failure: "2021-01-20T19:00:07.918151+0000 mds.d (mds.1) 2 : cluster [WRN] Scrub error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see mds.d log and `damage ls` output for details" in cluster log
2021-01-21T06:09:17.094 INFO:scrape:1 jobs: ['5805124']
2021-01-21T06:09:17.095 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}']
2021-01-21T06:09:17.095 INFO:scrape:
2021-01-21T06:09:17.095 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi104 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d8fc237ed7d71190da53e81746dff3e13ae8eb56 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2021-01-21T06:09:17.096 INFO:scrape:1 jobs: ['5805099']
2021-01-21T06:09:17.096 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/testing/{flavor/centos_latest', 'overrides/{frag_enable', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.097 INFO:scrape:
2021-01-21T06:09:17.097 INFO:scrape:Failure: Test failure: test_dispatch_queue_throttle_message (tasks.cephfs.test_misc.TestMisc)
2021-01-21T06:09:17.098 INFO:scrape:1 jobs: ['5805036']
2021-01-21T06:09:17.098 INFO:scrape:suites: ['clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'tasks/cephfs_misc_tests}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.098 INFO:scrape:
2021-01-21T06:09:17.099 INFO:scrape:Dead: b'2021-01-21T06:05:59.673 DEBUG:teuthology.exit:Finished running handlers'
2021-01-21T06:09:17.099 INFO:scrape:1 jobs: ['5805095']
2021-01-21T06:09:17.100 INFO:scrape:suites: ['2-workunit/cfuse_workunit_suites_fsstress}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/multifs/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.101 INFO:scrape:
2021-01-21T06:09:17.101 INFO:scrape:Failure: reached maximum tries (150) after waiting for 900 seconds
2021-01-21T06:09:17.101 INFO:scrape:1 jobs: ['5805115']
2021-01-21T06:09:17.102 INFO:scrape:suites: ['2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag_enable', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-21T06:09:17.103 INFO:scrape:
2021-01-21T06:09:17.109 INFO:teuthology.results:Sending results to jcollin@redhat.com: Test Run: jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/
failed:  21
dead:    1
running: 0
waiting: 0
queued:  0
passed:  78


Fail
=================================================================
[5805036]  fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:16:24
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805036/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805036/
sentry: https://sentry.ceph.com/organizations/ceph/?query=a6660cba90214e87aefec4fba473e499

    Test failure: test_dispatch_queue_throttle_message
    (tasks.cephfs.test_misc.TestMisc)

[5805048]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:36:36
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805048/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805048/

    "2021-01-20T18:03:20.077799+0000 mds.e (mds.0) 39 : cluster [WRN] Scrub
    error on inode 0x10000000263 (/client.0/tmp/testdir/dir1) see mds.e log and
    `damage ls` output for details" in cluster log

[5805051]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:35:08
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805051/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805051/

    "2021-01-20T17:43:27.889126+0000 osd.0 (osd.0) 3 : cluster [WRN] Throttler
    Limit has been hit. Some message processing may be significantly delayed."
    in cluster log

[5805055]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:52:04
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805055/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805055/

    "2021-01-20T18:07:32.922284+0000 mon.a (mon.0) 318 : cluster [WRN] Health
    check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in
    cluster log

[5805058]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:15:27
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805058/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805058/

    "2021-01-20T17:57:07.680391+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fb (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/ballista) see mds.f log and `damage ls` output for
    details" in cluster log

[5805067]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:38:44
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805067/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805067/

    "2021-01-20T17:58:57.009282+0000 osd.0 (osd.0) 3 : cluster [WRN] Throttler
    Limit has been hit. Some message processing may be significantly delayed."
    in cluster log

[5805069]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:11:59
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805069/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805069/

    "2021-01-20T18:12:26.613370+0000 mds.d (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.d log and `damage
    ls` output for details" in cluster log

[5805073]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:39:49
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805073/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805073/

    "2021-01-20T18:14:14.354939+0000 mds.d (mds.0) 69 : cluster [WRN] Scrub
    error on inode 0x1000000cd1e
    (/client.0/tmp/payload.1/multiple_rsync_payload.125785/python2.7/dist-
    packages/matplotlib) see mds.d log and `damage ls` output for details" in
    cluster log

[5805076]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:37:57
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805076/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805076/

    "2021-01-20T19:19:55.533058+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x10000002aa2
    (/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-59) see mds.f log and
    `damage ls` output for details" in cluster log

[5805079]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:47:32
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805079/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805079/

    "2021-01-20T18:11:45.154892+0000 mds.d (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.d log and `damage
    ls` output for details" in cluster log

[5805082]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:44:25
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805082/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805082/

    "2021-01-20T18:11:40.387206+0000 osd.1 (osd.1) 3 : cluster [WRN] Throttler
    Limit has been hit. Some message processing may be significantly delayed."
    in cluster log

[5805096]  fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:04:54
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805096/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805096/
sentry: https://sentry.ceph.com/organizations/ceph/?query=94ef64c057564022bbffef16fbe91574

    Command failed (workunit test suites/fsstress.sh) on smithi089 with status
    135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d8fc237ed7d71190da53e81746dff3e13ae8eb56
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[5805098]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:27:32
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805098/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805098/

    "2021-01-20T18:32:17.305868+0000 mds.e (mds.3) 2 : cluster [WRN] Scrub
    error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see
    mds.e log and `damage ls` output for details" in cluster log

[5805099]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:28:40
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805099/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805099/
sentry: https://sentry.ceph.com/organizations/ceph/?query=768b833f45764fc69b1a32dd937027b3

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi104
    with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=d8fc237ed7d71190da53e81746dff3e13ae8eb56
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[5805101]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:40:39
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805101/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805101/

    "2021-01-20T18:32:45.841472+0000 osd.0 (osd.0) 3 : cluster [WRN] Throttler
    Limit has been hit. Some message processing may be significantly delayed."
    in cluster log

[5805102]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:21:54
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805102/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805102/

    "2021-01-20T18:39:55.178695+0000 mds.a (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fa (/client.0/tmp/blogbench-1.0/man) see mds.a
    log and `damage ls` output for details" in cluster log

[5805109]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:05
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805109/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805109/

    "2021-01-20T18:58:19.123081+0000 mds.c (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000012df (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/open_posix_testsuite) see mds.c log and `damage ls`
    output for details" in cluster log

[5805115]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   02:38:55
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805115/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805115/

    reached maximum tries (150) after waiting for 900 seconds

[5805117]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:39:15
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805117/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805117/

    "2021-01-20T18:49:42.081918+0000 osd.0 (osd.0) 3 : cluster [WRN] Throttler
    Limit has been hit. Some message processing may be significantly delayed."
    in cluster log

[5805119]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:11:37
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805119/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805119/

    "2021-01-20T19:14:51.428563+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.f log and `damage
    ls` output for details" in cluster log

[5805124]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:31:17
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805124/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805124/

    "2021-01-20T19:00:07.918151+0000 mds.d (mds.1) 2 : cluster [WRN] Scrub
    error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see
    mds.d log and `damage ls` output for details" in cluster log



Dead
=================================================================
[5805095]  fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805095/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805095/



Pass
=================================================================
[5805031] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:13:55
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805031/

[5805032] fs/bugs/client_trim_caps/{begin clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i22073}
time:   00:07:08
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805032/

[5805033] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:09:54
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805033/

[5805034] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:07:29
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805034/

[5805035] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:27:28
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805035/

[5805037] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:39:31
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805037/

[5805038] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:08:29
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805038/

[5805039] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:32:16
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805039/

[5805040] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:07:11
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805040/

[5805041] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   02:26:58
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805041/

[5805042] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{centos_8} tasks/fstop}
time:   00:10:08
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805042/

[5805043] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:20:11
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805043/

[5805044] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   00:33:45
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805044/

[5805045] fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   00:54:51
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805045/

[5805046] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:47:25
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805046/

[5805047] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:42
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805047/

[5805049] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   02:30:31
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805049/

[5805050] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:38:04
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805050/

[5805052] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:19:37
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805052/

[5805053] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:58:00
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805053/

[5805054] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:19:45
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805054/

[5805056] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:53
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805056/

[5805057] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:15:54
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805057/

[5805059] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:12:20
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805059/

[5805060] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:38:30
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805060/

[5805061] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:16
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805061/

[5805062] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:26
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805062/

[5805063] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:12
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805063/

[5805064] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:12:41
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805064/

[5805065] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:24:45
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805065/

[5805066] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:53:32
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805066/

[5805068] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:10:06
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805068/

[5805070] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:14:25
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805070/

[5805071] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:57
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805071/

[5805072] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:19
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805072/

[5805074] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   01:54:49
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805074/

[5805075] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:18:30
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805075/

[5805077] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:14:32
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805077/

[5805078] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:57:57
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805078/

[5805080] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:01:12
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805080/

[5805081] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:13:48
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805081/

[5805083] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:15:02
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805083/

[5805084] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:13:51
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805084/

[5805085] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:28:21
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805085/

[5805086] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:18:43
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805086/

[5805087] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:43
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805087/

[5805088] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
time:   00:09:55
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805088/

[5805089] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:19:46
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805089/

[5805090] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:57:49
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805090/

[5805091] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:11:13
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805091/

[5805092] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:18:06
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805092/

[5805093] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
time:   00:13:18
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805093/

[5805094] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:13:14
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805094/

[5805097] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:29
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805097/

[5805100] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:50:47
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805100/

[5805103] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
time:   00:22:26
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805103/

[5805104] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:05:54
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805104/

[5805105] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:14:35
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805105/

[5805106] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:26:29
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805106/

[5805107] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:46
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805107/

[5805108] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:16:40
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805108/

[5805110] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:20:02
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805110/

[5805111] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:26:19
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805111/

[5805112] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:13:45
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805112/

[5805113] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:23
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805113/

[5805114] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:54
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805114/

[5805116] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:31:50
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805116/

[5805118] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:12:16
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805118/

[5805120] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:17:50
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805120/

[5805121] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:10:01
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805121/

[5805122] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:15:22
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805122/

[5805123] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:11
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805123/

[5805125] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   02:41:54
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805125/

[5805126] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:10:07
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805126/

[5805127] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:21:42
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805127/

[5805128] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:12:09
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805128/

[5805129] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:08:45
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805129/

[5805130] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:33:38
info:   http://pulpito.ceph.com/jcollin-2021-01-20_13:57:29-fs-wip-jcollin-dispatch-throttle-default-config-distro-basic-smithi/5805130/