2021-01-21T20:18:25.688 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-21T20:18:25.689 INFO:teuthology.results:seed: '1419'
2021-01-21T22:04:06.462 INFO:root:teuthology version: 1.0.0-fe79ba1b
2021-01-21T22:04:08.119 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2021-01-22T01:45:14.031 INFO:teuthology.results:Tests finished! gathering results...
2021-01-22T01:45:25.899 INFO:scrape:Found 100 jobs
2021-01-22T01:45:34.943 INFO:scrape:Found 7 distinct failure reasons
2021-01-22T01:45:34.944 INFO:scrape:Failure: "2021-01-21T20:56:41.021216+0000 mds.a (mds.2) 4 : cluster [WRN] Scrub error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see mds.a log and `damage ls` output for details" in cluster log
2021-01-22T01:45:34.944 INFO:scrape:8 jobs: ['5807722', '5807776', '5807732', '5807750', '5807783', '5807772', '5807743', '5807753']
2021-01-22T01:45:34.945 INFO:scrape:suites intersection: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag_enable', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:45:34.945 INFO:scrape:suites union: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}', 'workunit/suites/blogbench}}', 'workunit/suites/ffsb}}', 'workunit/suites/fsstress}}', 'workunit/suites/pjd}}']
2021-01-22T01:45:34.946 INFO:scrape:
2021-01-22T01:45:34.946 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=89cf4b38a9cb031e967fd5119f3aa0487be0c204 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'
2021-01-22T01:45:34.947 INFO:scrape:1 jobs: ['5807767']
2021-01-22T01:45:34.947 INFO:scrape:suites: ['2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:45:34.948 INFO:scrape:
2021-01-22T01:45:34.948 INFO:scrape:Failure: 502 Server Error: Bad Gateway for url: https://2.chacra.ceph.com/repos/ceph/wip-jcollin-dispatch-throttle-test-config-thrash-ffsb/89cf4b38a9cb031e967fd5119f3aa0487be0c204/ubuntu/bionic/flavors/default/repo
2021-01-22T01:45:34.948 INFO:scrape:4 jobs: ['5807762', '5807786', '5807778', '5807710']
2021-01-22T01:45:34.949 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag_enable', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:45:34.949 INFO:scrape:suites union: ['2-workunit/fs/trivial_sync}}', '2-workunit/suites/pjd}}', 'clusters/1-mds-2-client', 'clusters/1a5s-mds-1c-client', 'clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/multiclient/{begin', 'fs/thrash/workloads/{begin', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/fuse', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/1', 'ranks/5', 'scrub/no', 'session_timeout', 'tasks/cephfs_misc_tests}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/dbench}}']
2021-01-22T01:45:34.950 INFO:scrape:
2021-01-22T01:45:34.950 INFO:scrape:Crash: Command failed on smithi041 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2021-01-22T01:45:34.950 INFO:scrape:ceph version 16.0.0-9201-g89cf4b38 (89cf4b38a9cb031e967fd5119f3aa0487be0c204) pacific (dev)
 1: /lib64/libpthread.so.0(+0x12dc0) [0xf9bddc0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x50b9a23]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x27bbec) [0x50b9bec]
 6: (Monitor::~Monitor()+0xc2d) [0x3e879d]
 7: (Monitor::~Monitor()+0xd) [0x3e87fd]
 8: main()
 9: __libc_start_main()
 10: _start()
2021-01-22T01:45:34.951 INFO:scrape:1 jobs: ['5807770']
2021-01-22T01:45:34.951 INFO:scrape:suites: ['centos_latest', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/verify/{begin', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag_enable', 'ranks/3', 'session_timeout', 'tasks/fsstress', 'validater/valgrind}', 'whitelist_health', 'whitelist_wrongly_marked_down}']
2021-01-22T01:45:34.952 INFO:scrape:
2021-01-22T01:45:34.952 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh
2021-01-22T01:45:34.952 INFO:scrape:1 jobs: ['5807798']
2021-01-22T01:45:34.953 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/3', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}}']
2021-01-22T01:45:34.953 INFO:scrape:
2021-01-22T01:45:34.954 INFO:scrape:Failure: "2021-01-21T21:12:42.551931+0000 mon.c (mon.0) 275 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log
2021-01-22T01:45:34.954 INFO:scrape:1 jobs: ['5807729']
2021-01-22T01:45:34.954 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/workload/{begin', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag_enable', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/ffsb}}']
2021-01-22T01:45:34.955 INFO:scrape:
2021-01-22T01:45:34.955 INFO:scrape:Failure: The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'}
2021-01-22T01:45:34.956 INFO:scrape:1 jobs: ['5807740']
2021-01-22T01:45:34.956 INFO:scrape:suites: ['clusters/1a5s-mds-1c-client-3node', 'conf/{client', 'distro/{rhel_8}', 'fs/workload/{begin', 'k-testing}', 'mds', 'mon', 'mount/kclient/{mount', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/{flavor/ubuntu_latest', 'overrides/{frag_enable', 'ranks/5', 'scrub/yes', 'session_timeout', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iogen}}']
2021-01-22T01:45:34.956 INFO:scrape:
2021-01-22T01:45:34.961 INFO:teuthology.results:Sending results to jcollin@redhat.com: Test Run: jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi
=================================================================
info:    http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/
failed:  17
dead:    0
running: 0
waiting: 0
queued:  0
passed:  83


Fail
=================================================================
[5807710]  fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:02:35
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807710/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807710/
sentry: https://sentry.ceph.com/organizations/ceph/?query=977c563fc7224e05b28c49f41e2f07de

    502 Server Error: Bad Gateway for url:
    https://2.chacra.ceph.com/repos/ceph/wip-jcollin-dispatch-throttle-test-
    config-thrash-ffsb/89cf4b38a9cb031e967fd5119f3aa0487be0c204/ubuntu/bionic/f
    lavors/default/repo

[5807722]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:38:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807722/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807722/

    "2021-01-21T20:56:41.021216+0000 mds.a (mds.2) 4 : cluster [WRN] Scrub
    error on inode 0x10000000265 (/client.0/tmp/testdir/dir1/dir2/dir3) see
    mds.a log and `damage ls` output for details" in cluster log

[5807729]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:52:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807729/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807729/

    "2021-01-21T21:12:42.551931+0000 mon.c (mon.0) 275 : cluster [WRN] Health
    check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in
    cluster log

[5807732]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:55
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807732/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807732/

    "2021-01-21T21:43:49.312225+0000 mds.a (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000003b1 (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/commands) see mds.a log and `damage ls` output for
    details" in cluster log

[5807740]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
-----------------------------------------------------------------
time:   00:48:43
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807740/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807740/

    The following counters failed to be set on mds daemons: {'mds.imported',
    'mds.exported'}

[5807743]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:11:48
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807743/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807743/

    "2021-01-21T21:26:04.636110+0000 mds.e (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.e log and `damage
    ls` output for details" in cluster log

[5807750]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:36:37
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807750/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807750/

    "2021-01-21T22:06:52.537875+0000 mds.b (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.b
    log and `damage ls` output for details" in cluster log

[5807753]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:50:15
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807753/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807753/

    "2021-01-21T22:02:51.612600+0000 mds.f (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.f log and `damage
    ls` output for details" in cluster log

[5807762]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}}
-----------------------------------------------------------------
time:   00:02:48
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807762/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807762/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4cbcea6016ba45b88b70ff051dad5c62

    502 Server Error: Bad Gateway for url:
    https://2.chacra.ceph.com/repos/ceph/wip-jcollin-dispatch-throttle-test-
    config-thrash-ffsb/89cf4b38a9cb031e967fd5119f3aa0487be0c204/ubuntu/bionic/f
    lavors/default/repo

[5807767]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:13:08
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807767/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807767/
sentry: https://sentry.ceph.com/organizations/ceph/?query=384f806bef2d409cbedb3d3a8bf7310f

    Command failed (workunit test suites/pjd.sh) on smithi002 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=89cf4b38a9cb031e967fd5119f3aa0487be0c204
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

[5807770]  fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   01:33:23
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807770/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807770/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b31727073c8140889ac85839b60aeef5

    Command failed on smithi041 with status 1: 'sudo rm -rf --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp'

[5807772]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:30:05
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807772/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807772/

    "2021-01-21T21:54:43.064475+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub
    error on inode 0x10000000266 (/client.0/tmp/testdir/dir1/dir2/dir3/dir4)
    see mds.f log and `damage ls` output for details" in cluster log

[5807776]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
-----------------------------------------------------------------
time:   00:29:00
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807776/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807776/

    "2021-01-21T22:37:54.832245+0000 mds.e (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.e
    log and `damage ls` output for details" in cluster log

[5807778]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:02:24
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807778/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807778/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c8a0fe25476546eba933b2729884ef5c

    502 Server Error: Bad Gateway for url:
    https://2.chacra.ceph.com/repos/ceph/wip-jcollin-dispatch-throttle-test-
    config-thrash-ffsb/89cf4b38a9cb031e967fd5119f3aa0487be0c204/ubuntu/bionic/f
    lavors/default/repo

[5807783]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
-----------------------------------------------------------------
time:   00:14:56
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807783/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807783/

    "2021-01-21T22:21:28.876375+0000 mds.c (mds.0) 14 : cluster [WRN] Scrub
    error on inode 0x100000012df (/client.0/tmp/fsstress/ltp-
    full-20091231/testcases/open_posix_testsuite) see mds.c log and `damage ls`
    output for details" in cluster log

[5807786]  fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:02:46
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807786/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807786/
sentry: https://sentry.ceph.com/organizations/ceph/?query=58aaca911ff541caa71dac003adba0d1

    502 Server Error: Bad Gateway for url:
    https://2.chacra.ceph.com/repos/ceph/wip-jcollin-dispatch-throttle-test-
    config-thrash-ffsb/89cf4b38a9cb031e967fd5119f3aa0487be0c204/ubuntu/bionic/f
    lavors/default/repo

[5807798]  fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
-----------------------------------------------------------------
time:   03:20:50
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807798/
log:    http://qa-proxy.ceph.com/teuthology/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807798/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6ef2ee6e18ad4a628799c0a030c6878f

    Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi096 with
    status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=89cf4b38a9cb031e967fd5119f3aa0487be0c204
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo
    ne.client.0/qa/workunits/fs/misc/multiple_rsync.sh'



Pass
=================================================================
[5807704] fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:12:31
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807704/

[5807705] fs/bugs/client_trim_caps/{begin clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i22073}
time:   00:07:04
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807705/

[5807706] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls}
time:   00:08:50
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807706/

[5807707] fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client}
time:   00:07:11
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807707/

[5807708] fs/mirror/{begin cephfs-mirror/{one-per-cluster} clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror}
time:   00:48:19
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807708/

[5807709] fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone}
time:   01:26:46
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807709/

[5807711] fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover}
time:   00:39:58
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807711/

[5807712] fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:11:25
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807712/

[5807713] fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell}
time:   00:32:02
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807713/

[5807714] fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:02:09
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807714/

[5807715] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   03:58:22
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807715/

[5807716] fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{rhel_8} tasks/fstop}
time:   00:09:33
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807716/

[5807717] fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:17:07
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807717/

[5807718] fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}}
time:   00:31:46
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807718/

[5807719] fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep}
time:   01:00:31
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807719/

[5807720] fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}}
time:   00:45:55
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807720/

[5807721] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:13:01
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807721/

[5807723] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:10:32
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807723/

[5807724] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:37:28
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807724/

[5807725] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:37:13
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807725/

[5807726] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:19:48
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807726/

[5807727] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:44:11
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807727/

[5807728] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:20:57
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807728/

[5807730] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:32
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807730/

[5807731] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:15:34
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807731/

[5807733] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:12:14
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807733/

[5807734] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:19:17
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807734/

[5807735] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}}
time:   00:12:27
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807735/

[5807736] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:44
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807736/

[5807737] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:12
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807737/

[5807738] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   01:22:05
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807738/

[5807739] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin}
time:   00:24:30
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807739/

[5807741] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:37:08
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807741/

[5807742] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:10:19
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807742/

[5807744] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:15:09
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807744/

[5807745] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:14:03
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807745/

[5807746] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:09:47
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807746/

[5807747] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}}
time:   00:42:58
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807747/

[5807748] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   01:34:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807748/

[5807749] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:18:36
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807749/

[5807751] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:14:38
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807751/

[5807752] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   00:52:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807752/

[5807754] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
time:   01:35:40
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807754/

[5807755] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:24:09
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807755/

[5807756] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}}
time:   00:47:21
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807756/

[5807757] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool}
time:   00:15:23
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807757/

[5807758] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}}
time:   00:14:26
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807758/

[5807759] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:26:07
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807759/

[5807760] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}}
time:   00:18:56
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807760/

[5807761] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:53
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807761/

[5807763] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:19:30
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807763/

[5807764] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:56:40
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807764/

[5807765] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}}
time:   00:12:10
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807765/

[5807766] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:19:28
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807766/

[5807768] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:12:41
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807768/

[5807769] fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}}
time:   00:22:56
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807769/

[5807771] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:12:35
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807771/

[5807773] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
time:   01:01:26
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807773/

[5807774] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   00:51:30
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807774/

[5807775] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}
time:   00:38:49
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807775/

[5807777] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree}
time:   00:21:30
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807777/

[5807779] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:17:13
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807779/

[5807780] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}}
time:   00:29:20
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807780/

[5807781] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}}
time:   00:09:36
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807781/

[5807782] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}}
time:   00:16:45
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807782/

[5807784] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
time:   00:19:38
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807784/

[5807785] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}}
time:   00:31:38
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807785/

[5807787] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}}
time:   00:11:32
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807787/

[5807788] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}}
time:   00:23:30
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807788/

[5807789] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   02:44:30
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807789/

[5807790] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}}
time:   00:33:39
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807790/

[5807791] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}}
time:   00:47:55
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807791/

[5807792] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
time:   00:12:59
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807792/

[5807793] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}}
time:   00:11:44
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807793/

[5807794] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}}
time:   00:18:57
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807794/

[5807795] fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
time:   00:09:45
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807795/

[5807796] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}}
time:   00:15:16
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807796/

[5807797] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:10:24
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807797/

[5807799] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}}
time:   02:32:19
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807799/

[5807800] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}}
time:   00:11:31
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807800/

[5807801] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}}
time:   00:21:39
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807801/

[5807802] fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}}
time:   00:11:28
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807802/

[5807803] fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}}
time:   01:12:45
info:   http://pulpito.ceph.com/jcollin-2021-01-21_10:16:46-fs-wip-jcollin-dispatch-throttle-test-config-thrash-ffsb-distro-basic-smithi/5807803/